| 2025-01-07 17:41:11,073 INFO StreamThr :2201506 [internal.py:wandb_internal():85] W&B internal server running at pid: 2201506, started at: 2025-01-07 17:41:11.073069 |
| 2025-01-07 17:41:11,074 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 17:41:11,086 INFO WriterThread:2201506 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/run-you6ychq.wandb |
| 2025-01-07 17:41:11,086 DEBUG SenderThread:2201506 [sender.py:send():379] send: header |
| 2025-01-07 17:41:11,095 DEBUG SenderThread:2201506 [sender.py:send():379] send: run |
| 2025-01-07 17:41:12,353 INFO SenderThread:2201506 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files |
| 2025-01-07 17:41:12,353 INFO SenderThread:2201506 [sender.py:_start_run_threads():1188] run started: you6ychq with start time 1736289671.073577 |
| 2025-01-07 17:41:12,371 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 17:41:12,371 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 17:41:12,442 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 17:41:12,447 DEBUG HandlerThread:2201506 [system_info.py:__init__():26] System info init |
| 2025-01-07 17:41:12,447 DEBUG HandlerThread:2201506 [system_info.py:__init__():41] System info init done |
| 2025-01-07 17:41:12,447 INFO HandlerThread:2201506 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 17:41:12,447 INFO SystemMonitor:2201506 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 17:41:12,447 INFO HandlerThread:2201506 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 17:41:12,448 INFO SystemMonitor:2201506 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 17:41:12,448 INFO SystemMonitor:2201506 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 17:41:12,449 INFO SystemMonitor:2201506 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 17:41:12,449 INFO SystemMonitor:2201506 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 17:41:12,450 INFO SystemMonitor:2201506 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 17:41:12,460 DEBUG HandlerThread:2201506 [system_info.py:probe():152] Probing system |
| 2025-01-07 17:41:12,461 DEBUG HandlerThread:2201506 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 17:41:12,466 DEBUG HandlerThread:2201506 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 17:41:12,466 DEBUG HandlerThread:2201506 [system_info.py:probe():200] Probing system done |
| 2025-01-07 17:41:12,466 DEBUG HandlerThread:2201506 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T22:41:12.460572', 'startedAt': '2025-01-07T22:41:11.057653', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3936.501, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.982, 'min': 800.0, 'max': 4700.0}, {'current': 4499.954, 'min': 800.0, 'max': 4700.0}, {'current': 4500.008, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 769.9485206604004}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 17:41:12,466 INFO HandlerThread:2201506 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 17:41:12,466 INFO HandlerThread:2201506 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 17:41:12,466 DEBUG HandlerThread:2201506 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 17:41:12,472 DEBUG HandlerThread:2201506 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 17:41:12,472 DEBUG HandlerThread:2201506 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 17:41:12,537 DEBUG HandlerThread:2201506 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 17:41:12,538 INFO HandlerThread:2201506 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 17:41:12,541 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:41:12,541 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 17:41:12,541 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 17:41:12,541 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 17:41:12,776 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 17:41:12,777 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 17:41:12,777 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:41:12,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:41:12,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:12,780 INFO HandlerThread:2201506 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/tb" |
| save: true |
| } |
| control { |
| relay_id: "12c98893ea374f7aaf99019a1fd4c21a" |
| } |
| _info { |
| stream_id: "you6ychq" |
| } |
|
|
| 2025-01-07 17:41:12,780 DEBUG HandlerThread:2201506 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 17:41:12,781 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/3pylbhxc-wandb-metadata.json |
| 2025-01-07 17:41:12,845 INFO wandb-upload_2:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/jms3xgfv-diff.patch |
| 2025-01-07 17:41:12,852 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/7bm9mgli-code/FISH/eval_robot.py |
| 2025-01-07 17:41:12,876 DEBUG SenderThread:2201506 [sender.py:send():379] send: telemetry |
| 2025-01-07 17:41:12,876 DEBUG SenderThread:2201506 [sender.py:send():379] send: tbrecord |
| 2025-01-07 17:41:12,876 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:41:12,876 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 with policy live |
| 2025-01-07 17:41:13,070 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/4mk44jie-events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:41:13,355 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-metadata.json |
| 2025-01-07 17:41:13,355 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:41:13,355 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/code/FISH/eval_robot.py |
| 2025-01-07 17:41:13,355 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/requirements.txt |
| 2025-01-07 17:41:13,355 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:13,355 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/diff.patch |
| 2025-01-07 17:41:13,356 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/code/FISH |
| 2025-01-07 17:41:13,356 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/code |
| 2025-01-07 17:41:13,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:14,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:15,354 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:15,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:16,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:16,878 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:17,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:18,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:19,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:20,639 DEBUG SenderThread:2201506 [sender.py:send():379] send: config |
| 2025-01-07 17:41:20,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:21,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:22,689 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:22,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:23,356 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:23,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:24,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:25,357 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:25,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:26,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:27,358 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:27,777 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:41:27,777 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:41:27,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:27,913 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:28,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:29,358 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:29,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:30,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:31,359 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:31,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:32,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:33,346 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:33,360 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:33,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:34,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:35,360 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:35,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:36,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:37,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:38,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:38,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:39,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:40,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:41,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:42,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:41:42,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:41:42,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:43,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:43,869 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:44,363 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/config.yaml |
| 2025-01-07 17:41:44,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:45,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:46,782 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:47,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:48,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:49,155 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:49,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:50,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:51,364 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:41:51,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:52,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:53,783 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:54,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:41:54,784 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:55,784 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:56,784 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:57,777 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:41:57,777 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:41:57,825 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:58,785 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:59,786 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:41:59,870 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:00,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:01,296 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:42:01,296 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_d9dd74fcc70d62e9bbe0.mp4 with policy now |
| 2025-01-07 17:42:01,297 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:42:01,298 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:42:01,298 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:42:01,298 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:42:01,367 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:01,368 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/0_eval_0_d9dd74fcc70d62e9bbe0.mp4 |
| 2025-01-07 17:42:01,368 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:42:01,368 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval |
| 2025-01-07 17:42:01,368 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media |
| 2025-01-07 17:42:01,368 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos |
| 2025-01-07 17:42:01,565 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/5txo3ami-media/videos/eval/0_eval_0_d9dd74fcc70d62e9bbe0.mp4 |
| 2025-01-07 17:42:01,786 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:01,842 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/narutocu-media/videos/eval/0_eval_0_d9dd74fcc70d62e9bbe0.mp4 |
| 2025-01-07 17:42:02,786 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:03,368 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:03,786 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:04,786 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:05,005 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:05,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:06,786 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:07,369 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:07,805 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:08,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:09,370 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:09,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:10,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:10,806 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:11,371 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:42:11,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:12,450 DEBUG SystemMonitor:2201506 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 17:42:12,451 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:42:12,777 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:42:12,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:42:12,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:13,372 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:13,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:14,787 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:15,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:16,329 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:16,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:17,373 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:17,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:18,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:19,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:20,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:21,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:21,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:22,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:23,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:24,793 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:25,376 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:25,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:26,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:26,922 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:27,777 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:42:27,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:42:27,825 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:28,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:29,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:30,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:31,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:32,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:32,794 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:33,795 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:34,795 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:35,379 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:35,795 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:36,796 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:37,379 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:37,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:37,795 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:38,795 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:39,795 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:40,796 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:41,796 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:42,455 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:42:42,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:42:42,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:42:42,796 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:42,866 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:43,381 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:43,544 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:42:43,544 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_817e91198c35c162a7b6.mp4 with policy now |
| 2025-01-07 17:42:43,547 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:42:43,547 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:42:43,548 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:42:43,548 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:42:43,796 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:43,805 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/2dkgrhom-media/videos/eval/1_eval_1_817e91198c35c162a7b6.mp4 |
| 2025-01-07 17:42:44,382 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:42:44,382 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/1_eval_1_817e91198c35c162a7b6.mp4 |
| 2025-01-07 17:42:44,382 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval |
| 2025-01-07 17:42:44,682 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/zfrdp6qj-media/videos/eval/1_eval_1_817e91198c35c162a7b6.mp4 |
| 2025-01-07 17:42:44,798 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:45,382 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:45,799 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:46,807 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:47,802 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:48,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:48,808 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:49,384 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:49,806 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:50,805 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:51,384 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:42:51,805 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:52,805 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:53,808 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:54,780 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:42:54,807 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:55,807 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:56,807 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:57,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:42:57,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:42:57,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:58,807 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:59,807 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:42:59,922 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:00,809 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:01,214 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:43:01,215 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:43:01,215 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:43:01,388 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:43:01,388 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:43:01,662 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/cz2by9io-events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:43:01,837 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:02,808 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:03,388 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:03,816 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:04,813 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:05,389 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:05,762 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:05,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:06,817 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:07,390 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:07,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:08,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:09,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:10,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:11,391 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:11,749 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:11,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:12,452 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:43:12,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:43:12,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:43:12,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:13,818 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:14,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:15,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:16,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:16,901 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:17,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:18,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:19,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:20,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:21,820 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:22,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:22,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:23,819 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:24,820 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:25,820 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:26,820 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:27,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:43:27,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:43:27,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:27,852 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:28,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:29,822 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:30,821 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:31,826 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:32,845 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:32,845 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:43:32,846 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:43:32,846 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_999cf175c56114412850.mp4 with policy now |
| 2025-01-07 17:43:32,846 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:43:32,846 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:43:32,848 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:43:32,893 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:33,105 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/w7j2nt6x-media/videos/eval/2_eval_2_999cf175c56114412850.mp4 |
| 2025-01-07 17:43:33,398 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:33,398 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:43:33,398 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/2_eval_2_999cf175c56114412850.mp4 |
| 2025-01-07 17:43:33,399 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval |
| 2025-01-07 17:43:33,731 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/vqrkfokf-media/videos/eval/2_eval_2_999cf175c56114412850.mp4 |
| 2025-01-07 17:43:33,826 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:34,828 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:35,399 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:35,830 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:36,830 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:37,841 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:38,784 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:38,831 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:39,400 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:39,827 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:40,828 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:41,401 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:41,838 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:42,452 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:43:42,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:43:42,779 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:43:42,833 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:43,832 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:43,833 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:44,831 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:45,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:46,833 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:47,050 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:43:47,050 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:43:47,051 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:43:47,403 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:43:47,403 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:43:47,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:48,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:49,404 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:49,792 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:49,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:50,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:51,405 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:51,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:52,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:53,405 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:43:53,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:54,834 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:55,789 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:43:55,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:56,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:57,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:43:57,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:43:57,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:58,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:43:59,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:00,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:00,858 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:01,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:02,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:03,409 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:03,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:04,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:05,835 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:06,146 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:06,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:07,410 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:07,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:08,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:09,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:10,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:11,791 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:11,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:12,453 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:44:12,778 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:44:12,778 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:44:12,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:13,836 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:14,837 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:15,837 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:16,837 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:16,896 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:17,837 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:18,814 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:44:18,814 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_9c6c04d39b507ee63e06.mp4 with policy now |
| 2025-01-07 17:44:18,814 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:44:18,815 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:44:18,815 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:44:18,819 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:44:18,839 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:19,083 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/vty3lh3k-media/videos/eval/3_eval_3_9c6c04d39b507ee63e06.mp4 |
| 2025-01-07 17:44:19,414 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:19,414 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:44:19,414 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/3_eval_3_9c6c04d39b507ee63e06.mp4 |
| 2025-01-07 17:44:19,414 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval |
| 2025-01-07 17:44:19,726 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/bqu0ilyb-media/videos/eval/3_eval_3_9c6c04d39b507ee63e06.mp4 |
| 2025-01-07 17:44:19,850 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:20,847 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:21,415 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:21,847 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:22,552 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:22,856 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:23,415 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:23,848 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:24,850 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:25,854 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:26,853 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:27,417 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:27,781 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:44:27,782 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:44:27,855 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:27,934 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:28,850 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:29,103 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:44:29,103 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:44:29,105 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:44:29,417 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:44:29,418 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:44:29,687 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/lwk40kkv-events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:44:29,897 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:30,861 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:31,418 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:31,868 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:32,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:32,937 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:33,862 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:34,862 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:35,420 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:44:35,862 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:36,862 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:37,862 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:38,798 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:38,862 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:39,863 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:40,863 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:41,863 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:42,454 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:44:42,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:44:42,779 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:44:42,863 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:43,845 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:43,863 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:44,863 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:45,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:46,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:47,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:48,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:49,808 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:49,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:50,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:51,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:52,864 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:53,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:54,809 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:54,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:55,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:56,866 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:57,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:44:57,779 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:44:57,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:58,865 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:44:59,831 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:44:59,873 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:00,873 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:01,097 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:45:01,097 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_09bc5e6ad0d0988acb31.mp4 with policy now |
| 2025-01-07 17:45:01,101 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:45:01,102 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:45:01,102 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:01,103 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:01,357 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/w9unkm6b-media/videos/eval/4_eval_4_09bc5e6ad0d0988acb31.mp4 |
| 2025-01-07 17:45:01,429 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:01,429 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:45:01,429 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/4_eval_4_09bc5e6ad0d0988acb31.mp4 |
| 2025-01-07 17:45:01,429 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval |
| 2025-01-07 17:45:01,703 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/sis5wbn8-media/videos/eval/4_eval_4_09bc5e6ad0d0988acb31.mp4 |
| 2025-01-07 17:45:01,873 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:02,875 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:03,429 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:03,874 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:04,837 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:45:04,878 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:05,882 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:06,875 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:07,431 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:07,879 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:08,883 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:09,432 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:09,880 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:10,061 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:45:10,152 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:45:10,152 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:10,153 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:10,432 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:45:10,432 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:45:10,568 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 17:45:10,568 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 17:45:10,809 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:45:10,875 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:11,034 INFO wandb-upload_1:2201506 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp1yqpfzdl |
| 2025-01-07 17:45:11,216 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 17:45:11,225 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:45:11,433 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:11,433 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:45:11,433 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/table/eval/total_success_rate_ci_6_da430633f673220357fd.table.json |
| 2025-01-07 17:45:11,433 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 17:45:11,433 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media |
| 2025-01-07 17:45:11,434 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/table/eval |
| 2025-01-07 17:45:11,434 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/table |
| 2025-01-07 17:45:11,446 INFO SenderThread:2201506 [sender.py:send_request_log_artifact():1525] logged artifact run-you6ychq-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTk5MDEyOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTcxODg1NQ==', 'latestArtifact': None}} |
| 2025-01-07 17:45:11,446 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:45:11,446 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-07 17:45:11,446 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:45:11,446 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,447 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,447 DEBUG SenderThread:2201506 [sender.py:send():379] send: telemetry |
| 2025-01-07 17:45:11,447 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,447 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,447 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,448 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,448 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,448 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,448 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,449 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,449 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,449 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,449 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,449 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,450 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,450 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,450 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,450 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,451 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,451 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,451 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,451 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,451 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:11,452 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:11,452 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 17:45:11,634 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/6tsaj4d8-media/table/eval/total_success_rate_ci_6_da430633f673220357fd.table.json |
| 2025-01-07 17:45:11,665 INFO wandb-upload_2:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/95aelmml-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 17:45:11,783 INFO wandb-upload_1:2201506 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmplhmw4f05 |
| 2025-01-07 17:45:11,875 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:11,933 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/oo6ga3vj-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 17:45:12,160 INFO SenderThread:2201506 [sender.py:send_request_log_artifact():1525] logged artifact run-you6ychq-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTk5MDIxMg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTcxODkyMA==', 'latestArtifact': None}} |
| 2025-01-07 17:45:12,160 DEBUG SenderThread:2201506 [sender.py:send():379] send: files |
| 2025-01-07 17:45:12,160 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_da430633f673220357fd.table.json with policy now |
| 2025-01-07 17:45:12,160 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:45:12,161 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:12,161 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:12,407 INFO wandb-upload_2:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/ssdkifkk-media/table/eval/total_success_rate_ci_6_da430633f673220357fd.table.json |
| 2025-01-07 17:45:12,433 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:45:12,454 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:45:12,779 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:45:12,779 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:45:12,868 DEBUG SenderThread:2201506 [sender.py:send():379] send: exit |
| 2025-01-07 17:45:12,868 INFO SenderThread:2201506 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 17:45:12,868 INFO SenderThread:2201506 [sender.py:send_exit():588] handling runtime: 240 |
| 2025-01-07 17:45:12,869 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:12,869 INFO SenderThread:2201506 [sender.py:send_exit():594] send defer |
| 2025-01-07 17:45:12,869 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:12,869 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 17:45:12,869 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:12,869 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 17:45:12,869 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 17:45:12,869 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:12,869 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 17:45:12,870 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:12,870 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 17:45:12,870 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 17:45:12,870 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:12,870 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 17:45:12,870 INFO HandlerThread:2201506 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 17:45:12,870 INFO HandlerThread:2201506 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 17:45:12,871 INFO HandlerThread:2201506 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 17:45:12,871 DEBUG SystemMonitor:2201506 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 17:45:12,871 DEBUG SystemMonitor:2201506 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 17:45:12,921 INFO HandlerThread:2201506 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 17:45:12,922 INFO HandlerThread:2201506 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 17:45:12,922 INFO HandlerThread:2201506 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 17:45:12,922 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:12,922 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 17:45:12,922 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 17:45:12,922 DEBUG SenderThread:2201506 [sender.py:send():379] send: stats |
| 2025-01-07 17:45:12,923 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:12,923 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 17:45:12,923 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:12,923 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 17:45:12,923 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 17:45:12,923 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:12,923 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 17:45:13,434 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:13,434 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:45:15,435 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:19,158 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:45:19,159 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:45:19,159 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 17:45:19,160 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:19,160 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 17:45:19,160 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 17:45:19,160 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:45:19,160 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:19,160 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 17:45:19,160 DEBUG SenderThread:2201506 [sender.py:send():379] send: history |
| 2025-01-07 17:45:19,161 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:45:19,161 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:19,161 DEBUG SenderThread:2201506 [sender.py:send():379] send: summary |
| 2025-01-07 17:45:19,165 INFO SenderThread:2201506 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:45:19,165 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:19,166 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 17:45:19,166 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 17:45:19,166 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:19,166 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 17:45:19,166 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:19,166 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 17:45:19,169 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:45:19,270 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 17:45:19,270 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:19,270 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 17:45:19,270 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:19,270 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 17:45:19,437 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/config.yaml |
| 2025-01-07 17:45:19,437 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:45:19,827 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:45:21,361 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 17:45:21,361 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:45:21,361 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:21,361 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 17:45:21,362 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:21,362 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 17:45:21,362 INFO SenderThread:2201506 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 17:45:21,364 INFO SenderThread:2201506 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 17:45:21,366 INFO SenderThread:2201506 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 17:45:21,380 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 17:45:21,380 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:21,380 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 17:45:21,381 DEBUG SenderThread:2201506 [sender.py:send():379] send: artifact |
| 2025-01-07 17:45:21,437 INFO Thread-12 :2201506 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:21,827 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:45:21,882 INFO SenderThread:2201506 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'versionIndex': 436}}} |
| 2025-01-07 17:45:21,882 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:21,882 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 17:45:21,882 INFO SenderThread:2201506 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 17:45:22,438 INFO SenderThread:2201506 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files |
| 2025-01-07 17:45:22,438 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/config.yaml config.yaml |
| 2025-01-07 17:45:22,438 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 17:45:22,438 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/requirements.txt requirements.txt |
| 2025-01-07 17:45:22,439 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 17:45:22,439 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:45:22,442 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log output.log |
| 2025-01-07 17:45:22,442 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/diff.patch diff.patch |
| 2025-01-07 17:45:22,444 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 17:45:22,444 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/1_eval_1_817e91198c35c162a7b6.mp4 media/videos/eval/1_eval_1_817e91198c35c162a7b6.mp4 |
| 2025-01-07 17:45:22,444 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/4_eval_4_09bc5e6ad0d0988acb31.mp4 media/videos/eval/4_eval_4_09bc5e6ad0d0988acb31.mp4 |
| 2025-01-07 17:45:22,444 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/0_eval_0_d9dd74fcc70d62e9bbe0.mp4 media/videos/eval/0_eval_0_d9dd74fcc70d62e9bbe0.mp4 |
| 2025-01-07 17:45:22,447 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/3_eval_3_9c6c04d39b507ee63e06.mp4 media/videos/eval/3_eval_3_9c6c04d39b507ee63e06.mp4 |
| 2025-01-07 17:45:22,447 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/videos/eval/2_eval_2_999cf175c56114412850.mp4 media/videos/eval/2_eval_2_999cf175c56114412850.mp4 |
| 2025-01-07 17:45:22,447 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/table/eval/total_success_rate_ci_6_da430633f673220357fd.table.json media/table/eval/total_success_rate_ci_6_da430633f673220357fd.table.json |
| 2025-01-07 17:45:22,448 INFO SenderThread:2201506 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-07 17:45:22,448 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 17:45:22,448 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:45:22,448 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:22,448 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 17:45:22,448 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:22,448 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 17:45:22,448 INFO SenderThread:2201506 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 17:45:22,613 INFO wandb-upload_0:2201506 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/config.yaml |
| 2025-01-07 17:45:22,622 INFO wandb-upload_1:2201506 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/wandb-summary.json |
| 2025-01-07 17:45:22,636 INFO wandb-upload_2:2201506 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/requirements.txt |
| 2025-01-07 17:45:22,655 INFO wandb-upload_4:2201506 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/files/output.log |
| 2025-01-07 17:45:22,799 INFO wandb-upload_3:2201506 [upload_job.py:push():130] Uploaded file /tmp/tmphvtllm8lwandb/qs52s3u7-events.out.tfevents.1736289672.leonmkim-ROG-Strix-G15CS-G15CS.2201339.0 |
| 2025-01-07 17:45:22,828 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:45:22,828 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:45:23,000 INFO Thread-11 (_thread_body):2201506 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 17:45:23,000 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:23,000 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 17:45:23,000 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:23,000 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 17:45:23,000 INFO SenderThread:2201506 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 17:45:23,001 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 17:45:23,002 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:23,002 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 17:45:23,002 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:23,002 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 17:45:23,002 INFO SenderThread:2201506 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 17:45:23,398 INFO SenderThread:2201506 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 17:45:23,398 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 17:45:23,399 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:23,399 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 17:45:23,399 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:23,399 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 17:45:23,399 INFO SenderThread:2201506 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 17:45:23,399 DEBUG SenderThread:2201506 [sender.py:send():379] send: final |
| 2025-01-07 17:45:23,399 DEBUG SenderThread:2201506 [sender.py:send():379] send: footer |
| 2025-01-07 17:45:23,399 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:45:23,399 INFO HandlerThread:2201506 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 17:45:23,400 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:45:23,400 INFO SenderThread:2201506 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 17:45:23,400 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:45:23,400 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:45:23,400 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 17:45:23,401 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 17:45:23,402 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:45:23,402 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 17:45:23,403 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 17:45:23,403 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:45:23,465 DEBUG SenderThread:2201506 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:45:23,465 INFO MainThread:2201506 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 17:45:23,465 INFO MainThread:2201506 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 17:45:23,466 INFO MainThread:2201506 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 17:45:23,466 DEBUG HandlerThread:2201506 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 17:45:23,466 INFO HandlerThread:2201506 [handler.py:finish():882] shutting down handler |
| 2025-01-07 17:45:24,402 INFO WriterThread:2201506 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/you6ychq/wandb/run-20250107_174111-you6ychq/run-you6ychq.wandb |
| 2025-01-07 17:45:24,465 INFO SenderThread:2201506 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 17:45:24,465 INFO SenderThread:2201506 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 17:45:24,465 INFO SenderThread:2201506 [file_pusher.py:join():175] waiting for file pusher |
|
|
