| 2025-01-07 20:21:14,147 INFO StreamThr :2224087 [internal.py:wandb_internal():85] W&B internal server running at pid: 2224087, started at: 2025-01-07 20:21:14.146861 |
| 2025-01-07 20:21:14,149 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 20:21:14,160 INFO WriterThread:2224087 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/run-w4b02c7s.wandb |
| 2025-01-07 20:21:14,161 DEBUG SenderThread:2224087 [sender.py:send():379] send: header |
| 2025-01-07 20:21:14,168 DEBUG SenderThread:2224087 [sender.py:send():379] send: run |
| 2025-01-07 20:21:14,462 INFO SenderThread:2224087 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files |
| 2025-01-07 20:21:14,462 INFO SenderThread:2224087 [sender.py:_start_run_threads():1188] run started: w4b02c7s with start time 1736299274.148413 |
| 2025-01-07 20:21:14,479 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 20:21:14,480 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 20:21:14,555 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 20:21:14,560 DEBUG HandlerThread:2224087 [system_info.py:__init__():26] System info init |
| 2025-01-07 20:21:14,560 DEBUG HandlerThread:2224087 [system_info.py:__init__():41] System info init done |
| 2025-01-07 20:21:14,560 INFO HandlerThread:2224087 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 20:21:14,561 INFO SystemMonitor:2224087 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 20:21:14,561 INFO HandlerThread:2224087 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 20:21:14,561 INFO SystemMonitor:2224087 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 20:21:14,562 INFO SystemMonitor:2224087 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 20:21:14,563 INFO SystemMonitor:2224087 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 20:21:14,563 INFO SystemMonitor:2224087 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 20:21:14,564 INFO SystemMonitor:2224087 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 20:21:14,573 DEBUG HandlerThread:2224087 [system_info.py:probe():152] Probing system |
| 2025-01-07 20:21:14,575 DEBUG HandlerThread:2224087 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 20:21:14,581 DEBUG HandlerThread:2224087 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 20:21:14,581 DEBUG HandlerThread:2224087 [system_info.py:probe():200] Probing system done |
| 2025-01-07 20:21:14,581 DEBUG HandlerThread:2224087 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T01:21:14.573756', 'startedAt': '2025-01-08T01:21:14.136932', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3935.56425, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4494.112, 'min': 800.0, 'max': 4700.0}, {'current': 4499.827, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 572.616024017334}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 20:21:14,581 INFO HandlerThread:2224087 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 20:21:14,581 INFO HandlerThread:2224087 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 20:21:14,581 DEBUG HandlerThread:2224087 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 20:21:14,584 DEBUG HandlerThread:2224087 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 20:21:14,584 DEBUG HandlerThread:2224087 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 20:21:14,650 DEBUG HandlerThread:2224087 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 20:21:14,651 INFO HandlerThread:2224087 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 20:21:14,654 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:21:14,654 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 20:21:14,654 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 20:21:14,654 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 20:21:14,876 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/1svrxmfh-wandb-metadata.json |
| 2025-01-07 20:21:14,891 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 20:21:14,891 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 20:21:14,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:14,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:21:14,892 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:21:14,907 INFO wandb-upload_2:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/yhj813vc-diff.patch |
| 2025-01-07 20:21:14,927 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/w5iryu7n-code/FISH/eval_robot.py |
| 2025-01-07 20:21:14,933 INFO HandlerThread:2224087 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/tb" |
| save: true |
| } |
| control { |
| relay_id: "426b523202ad4154bb07363de3b5e08c" |
| } |
| _info { |
| stream_id: "w4b02c7s" |
| } |
|
|
| 2025-01-07 20:21:14,933 DEBUG HandlerThread:2224087 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 20:21:14,945 DEBUG SenderThread:2224087 [sender.py:send():379] send: telemetry |
| 2025-01-07 20:21:14,998 DEBUG SenderThread:2224087 [sender.py:send():379] send: tbrecord |
| 2025-01-07 20:21:14,999 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:21:14,999 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 with policy live |
| 2025-01-07 20:21:15,173 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/wscof4s7-events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:21:15,464 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-metadata.json |
| 2025-01-07 20:21:15,464 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:21:15,464 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/requirements.txt |
| 2025-01-07 20:21:15,464 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/code/FISH/eval_robot.py |
| 2025-01-07 20:21:15,464 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/diff.patch |
| 2025-01-07 20:21:15,465 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:15,465 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/code |
| 2025-01-07 20:21:15,465 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/code/FISH |
| 2025-01-07 20:21:15,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:16,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:17,463 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:17,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:18,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:19,350 DEBUG SenderThread:2224087 [sender.py:send():379] send: config |
| 2025-01-07 20:21:19,350 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:19,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:20,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:21,464 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:21,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:22,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:23,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:24,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:24,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:25,465 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:25,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:26,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:27,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:28,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:29,467 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:29,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:21:29,892 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:21:29,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:30,036 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:30,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:31,467 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:31,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:32,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:33,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:34,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:35,468 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:35,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:35,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:36,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:37,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:38,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:39,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:40,893 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:40,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:41,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:42,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:43,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:44,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:21:44,892 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:21:44,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:45,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:46,395 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:46,615 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:21:46,616 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_bec6c78d6d5d8da97ef5.mp4 with policy now |
| 2025-01-07 20:21:46,616 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:21:46,617 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:21:46,617 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:21:46,619 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:21:46,851 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/brgutril-media/videos/eval/0_eval_0_bec6c78d6d5d8da97ef5.mp4 |
| 2025-01-07 20:21:46,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/config.yaml |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/0_eval_0_bec6c78d6d5d8da97ef5.mp4 |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos |
| 2025-01-07 20:21:47,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval |
| 2025-01-07 20:21:47,743 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/aawlbvhc-media/videos/eval/0_eval_0_bec6c78d6d5d8da97ef5.mp4 |
| 2025-01-07 20:21:47,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:48,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:49,476 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:49,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:50,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:51,477 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:51,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:51,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:52,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:53,478 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:21:53,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:54,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:55,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:56,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:57,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:21:57,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:58,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:21:59,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:21:59,892 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:21:59,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:00,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:01,480 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:01,480 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:22:01,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:02,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:03,481 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:03,585 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:03,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:04,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:05,482 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:05,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:06,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:07,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:08,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:08,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:09,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:10,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:11,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:12,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:13,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:13,900 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:14,564 DEBUG SystemMonitor:2224087 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 20:22:14,564 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:22:14,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:22:14,894 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:22:14,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:15,900 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:16,900 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:17,900 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:18,900 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:19,841 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:19,900 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:20,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:21,487 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:21,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:22,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:23,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:24,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:24,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:25,489 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:25,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:26,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:27,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:28,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:29,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:22:29,894 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:22:29,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:30,045 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:30,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:31,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:32,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:33,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:34,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:35,894 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:35,903 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:36,903 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:37,903 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:38,903 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:39,903 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:40,245 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:22:40,248 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:22:40,248 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_407b26ce0b9569f5bbfd.mp4 with policy now |
| 2025-01-07 20:22:40,248 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:22:40,248 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:22:40,249 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:22:40,494 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:22:40,494 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/1_eval_1_407b26ce0b9569f5bbfd.mp4 |
| 2025-01-07 20:22:40,495 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval |
| 2025-01-07 20:22:40,520 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/i8mv5ivj-media/videos/eval/1_eval_1_407b26ce0b9569f5bbfd.mp4 |
| 2025-01-07 20:22:40,806 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/nn13y66a-media/videos/eval/1_eval_1_407b26ce0b9569f5bbfd.mp4 |
| 2025-01-07 20:22:40,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:41,249 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:41,495 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:41,904 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:42,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:43,906 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:44,565 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:22:44,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:22:44,901 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:22:44,945 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:45,496 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:45,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:46,669 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:46,905 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:47,904 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:48,905 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:49,498 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:49,905 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:50,905 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:51,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:51,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:52,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:53,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:54,338 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:22:54,338 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:22:54,339 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:22:54,499 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:22:54,499 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:22:54,739 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/0yikza4i-events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:22:54,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:55,500 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:55,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:56,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:22:56,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:57,500 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:57,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:58,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:22:59,501 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:22:59,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:22:59,896 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:22:59,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:00,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:01,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:02,003 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:02,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:03,503 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:03,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:04,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:05,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:06,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:07,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:07,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:08,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:09,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:10,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:11,910 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:12,895 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:12,910 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:13,910 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:14,565 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:23:14,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:23:14,896 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:23:14,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:15,910 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:16,910 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:17,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:18,003 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:18,910 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:19,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:20,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:21,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:22,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:23,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:23,911 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:24,912 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:25,912 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:26,912 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:27,912 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:28,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:28,912 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:29,896 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:23:29,897 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:23:29,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:30,912 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:31,913 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:32,921 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:33,458 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:23:33,458 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_a036985f900b09e578db.mp4 with policy now |
| 2025-01-07 20:23:33,471 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:23:33,472 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:23:33,472 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:23:33,476 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:23:33,513 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:33,513 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:23:33,513 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/2_eval_2_a036985f900b09e578db.mp4 |
| 2025-01-07 20:23:33,513 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval |
| 2025-01-07 20:23:33,709 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/v51k45zd-media/videos/eval/2_eval_2_a036985f900b09e578db.mp4 |
| 2025-01-07 20:23:33,914 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:33,990 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/js1noyat-media/videos/eval/2_eval_2_a036985f900b09e578db.mp4 |
| 2025-01-07 20:23:34,511 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:34,916 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:35,513 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:35,921 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:36,925 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:37,922 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:38,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:39,515 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:39,905 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:39,924 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:40,925 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:41,516 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:41,929 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:42,928 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:43,927 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:44,566 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:23:44,901 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:23:44,901 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:23:44,945 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:44,982 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:45,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:46,941 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:47,931 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:48,182 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:23:48,182 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:23:48,184 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:23:48,518 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:23:48,518 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:23:48,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:49,518 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:49,944 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:50,905 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:50,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:51,519 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:51,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:52,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:53,520 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:23:53,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:54,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:55,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:56,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:23:56,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:57,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:58,933 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:23:59,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:23:59,897 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:23:59,941 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:00,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:01,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:02,018 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:02,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:03,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:04,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:05,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:06,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:07,245 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:07,524 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:07,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:08,934 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:09,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:10,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:11,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:12,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:12,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:13,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:14,566 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:24:14,897 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:24:14,898 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:24:14,941 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:15,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:16,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:17,935 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:17,958 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:18,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:19,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:20,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:21,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:22,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:23,302 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:23,530 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:23,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:24,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:25,936 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:26,937 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:27,532 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:27,880 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:24:27,880 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_c5460bdfba046140b658.mp4 with policy now |
| 2025-01-07 20:24:27,888 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:24:27,888 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:24:27,888 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:24:27,895 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:24:27,942 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:28,173 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/xg728rec-media/videos/eval/3_eval_3_c5460bdfba046140b658.mp4 |
| 2025-01-07 20:24:28,532 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:24:28,532 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/3_eval_3_c5460bdfba046140b658.mp4 |
| 2025-01-07 20:24:28,532 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval |
| 2025-01-07 20:24:28,833 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/20hvtfdl-media/videos/eval/3_eval_3_c5460bdfba046140b658.mp4 |
| 2025-01-07 20:24:28,915 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:28,944 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:29,532 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:29,899 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:24:29,899 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:24:29,943 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:30,957 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:31,954 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:32,951 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:33,534 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:33,953 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:34,418 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:34,958 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:35,534 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:35,958 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:36,953 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:37,960 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:38,964 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:39,902 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:39,967 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:40,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:41,963 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:42,537 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:24:42,815 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/zr5gfofa-events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:24:42,971 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:43,247 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:24:43,247 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:24:43,247 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:24:43,537 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:43,537 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:24:43,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:44,567 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:24:44,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:24:44,898 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:24:44,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:45,043 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:45,538 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:45,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:46,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:47,539 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:47,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:48,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:49,540 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:24:49,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:50,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:50,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:51,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:52,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:53,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:54,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:55,965 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:56,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:24:56,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:57,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:58,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:24:59,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:24:59,898 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:24:59,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:00,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:01,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:01,971 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:02,966 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:03,967 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:04,967 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:05,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:06,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:07,907 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:07,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:08,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:09,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:10,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:11,547 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:11,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:12,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:13,017 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:13,548 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:13,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:14,567 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:25:14,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:25:14,898 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:25:14,968 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:15,549 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:15,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:16,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:17,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:18,050 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:18,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:19,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:20,969 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:21,974 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:22,780 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:25:22,780 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_dc542bb6f251b5d41e0c.mp4 with policy now |
| 2025-01-07 20:25:22,781 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:25:22,782 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:25:22,782 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:22,801 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:22,973 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:23,069 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/28l1v0g1-media/videos/eval/4_eval_4_dc542bb6f251b5d41e0c.mp4 |
| 2025-01-07 20:25:23,551 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:23,551 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:25:23,552 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/4_eval_4_dc542bb6f251b5d41e0c.mp4 |
| 2025-01-07 20:25:23,552 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval |
| 2025-01-07 20:25:23,807 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:23,880 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/q1iq46wv-media/videos/eval/4_eval_4_dc542bb6f251b5d41e0c.mp4 |
| 2025-01-07 20:25:23,976 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:24,981 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:25,552 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:25,982 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:27,001 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:27,553 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:27,987 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:28,980 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:29,321 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:29,908 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:25:29,908 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:25:29,979 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:30,982 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:31,554 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:31,984 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:32,992 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:33,992 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:34,922 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:34,995 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:36,002 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:37,002 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:37,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:38,072 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 20:25:38,072 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 20:25:38,230 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:25:38,522 INFO wandb-upload_1:2224087 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmprnm7pcgi |
| 2025-01-07 20:25:38,557 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:25:38,557 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table/eval/success_rate_ci_5_8490adedbf62c026e308.table.json |
| 2025-01-07 20:25:38,557 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media |
| 2025-01-07 20:25:38,557 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table/eval |
| 2025-01-07 20:25:38,557 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table |
| 2025-01-07 20:25:38,663 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 20:25:38,676 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 20:25:38,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:39,075 INFO SenderThread:2224087 [sender.py:send_request_log_artifact():1525] logged artifact run-w4b02c7s-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjM4Mzc5MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTgzMTIwMg==', 'latestArtifact': None}} |
| 2025-01-07 20:25:39,075 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:25:39,075 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_8490adedbf62c026e308.table.json with policy now |
| 2025-01-07 20:25:39,075 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:25:39,075 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,076 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,076 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:25:39,076 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,080 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,080 DEBUG SenderThread:2224087 [sender.py:send():379] send: telemetry |
| 2025-01-07 20:25:39,080 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,081 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,081 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,081 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,081 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,082 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,082 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,082 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,082 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,083 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,083 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,083 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,083 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,083 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,084 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,084 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,084 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,084 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,085 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,085 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,085 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,085 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,085 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 20:25:39,251 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/tuf9qeia-media/table/eval/success_rate_ci_5_8490adedbf62c026e308.table.json |
| 2025-01-07 20:25:39,481 INFO wandb-upload_0:2224087 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpqcd0wrrf |
| 2025-01-07 20:25:39,503 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/d6bqe7eu-media/table/eval/success_rate_ci_5_8490adedbf62c026e308.table.json |
| 2025-01-07 20:25:39,557 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:39,558 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:25:39,558 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:25:39,558 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table/eval/total_success_rate_ci_6_579a029f256e694027d0.table.json |
| 2025-01-07 20:25:39,559 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table/eval |
| 2025-01-07 20:25:39,863 INFO SenderThread:2224087 [sender.py:send_request_log_artifact():1525] logged artifact run-w4b02c7s-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjM4MzgxNg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTgzMTIwMw==', 'latestArtifact': None}} |
| 2025-01-07 20:25:39,863 DEBUG SenderThread:2224087 [sender.py:send():379] send: files |
| 2025-01-07 20:25:39,863 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_579a029f256e694027d0.table.json with policy now |
| 2025-01-07 20:25:39,865 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:25:39,865 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:25:39,866 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:39,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:40,049 INFO wandb-upload_2:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/8mliszsk-media/table/eval/total_success_rate_ci_6_579a029f256e694027d0.table.json |
| 2025-01-07 20:25:40,306 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/9ys5wxb3-media/table/eval/total_success_rate_ci_6_579a029f256e694027d0.table.json |
| 2025-01-07 20:25:40,558 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:25:40,916 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:40,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:41,558 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:25:41,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:42,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:43,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:44,568 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:25:44,898 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 20:25:44,898 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 20:25:44,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:45,978 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:45,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:46,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:47,997 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:48,998 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:49,998 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:50,998 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:51,916 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:25:51,998 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:52,998 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:53,998 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:25:54,889 DEBUG SenderThread:2224087 [sender.py:send():379] send: exit |
| 2025-01-07 20:25:54,889 INFO SenderThread:2224087 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 20:25:54,889 INFO SenderThread:2224087 [sender.py:send_exit():588] handling runtime: 280 |
| 2025-01-07 20:25:54,890 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:25:54,890 INFO SenderThread:2224087 [sender.py:send_exit():594] send defer |
| 2025-01-07 20:25:54,890 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:25:54,890 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 20:25:54,890 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:25:54,890 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 20:25:54,890 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 20:25:54,890 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:25:54,890 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 20:25:54,891 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:25:54,891 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 20:25:54,891 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 20:25:54,891 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:25:54,891 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 20:25:54,891 INFO HandlerThread:2224087 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 20:25:54,891 DEBUG SystemMonitor:2224087 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 20:25:54,891 DEBUG SystemMonitor:2224087 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 20:25:54,892 INFO HandlerThread:2224087 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 20:25:54,892 INFO HandlerThread:2224087 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 20:25:54,945 INFO HandlerThread:2224087 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 20:25:54,945 INFO HandlerThread:2224087 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 20:25:54,945 INFO HandlerThread:2224087 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 20:25:54,946 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:25:54,946 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 20:25:54,946 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 20:25:54,946 DEBUG SenderThread:2224087 [sender.py:send():379] send: stats |
| 2025-01-07 20:25:54,946 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:25:54,946 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 20:25:54,946 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:25:54,946 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 20:25:54,946 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 20:25:54,947 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:25:54,947 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 20:25:55,565 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:25:57,566 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:26:01,320 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 20:26:01,321 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:26:01,321 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 20:26:01,321 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:01,322 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 20:26:01,322 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 20:26:01,322 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 20:26:01,322 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:01,322 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 20:26:01,323 DEBUG SenderThread:2224087 [sender.py:send():379] send: history |
| 2025-01-07 20:26:01,323 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 20:26:01,323 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:26:01,324 DEBUG SenderThread:2224087 [sender.py:send():379] send: summary |
| 2025-01-07 20:26:01,327 INFO SenderThread:2224087 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 20:26:01,328 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:01,328 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 20:26:01,328 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 20:26:01,328 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:01,328 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 20:26:01,328 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:01,328 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 20:26:01,331 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 20:26:01,445 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 20:26:01,445 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:01,445 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 20:26:01,445 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:01,446 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 20:26:01,567 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/config.yaml |
| 2025-01-07 20:26:01,567 INFO Thread-12 :2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:26:01,890 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 20:26:04,759 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 20:26:04,759 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 20:26:04,760 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:04,760 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 20:26:04,760 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:04,760 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 20:26:04,760 INFO SenderThread:2224087 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 20:26:04,762 INFO SenderThread:2224087 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 20:26:04,764 INFO SenderThread:2224087 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 20:26:04,779 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 20:26:04,779 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:04,779 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 20:26:04,779 DEBUG SenderThread:2224087 [sender.py:send():379] send: artifact |
| 2025-01-07 20:26:04,891 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 20:26:04,993 INFO SenderThread:2224087 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTc5MjM4Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjE0MDMwNQ==', 'versionIndex': 437}}} |
| 2025-01-07 20:26:04,993 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:04,993 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 20:26:04,993 INFO SenderThread:2224087 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 20:26:05,568 INFO SenderThread:2224087 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/config.yaml config.yaml |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/requirements.txt requirements.txt |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log output.log |
| 2025-01-07 20:26:05,569 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/diff.patch diff.patch |
| 2025-01-07 20:26:05,570 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 20:26:05,573 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/4_eval_4_dc542bb6f251b5d41e0c.mp4 media/videos/eval/4_eval_4_dc542bb6f251b5d41e0c.mp4 |
| 2025-01-07 20:26:05,573 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/1_eval_1_407b26ce0b9569f5bbfd.mp4 media/videos/eval/1_eval_1_407b26ce0b9569f5bbfd.mp4 |
| 2025-01-07 20:26:05,573 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/0_eval_0_bec6c78d6d5d8da97ef5.mp4 media/videos/eval/0_eval_0_bec6c78d6d5d8da97ef5.mp4 |
| 2025-01-07 20:26:05,573 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/3_eval_3_c5460bdfba046140b658.mp4 media/videos/eval/3_eval_3_c5460bdfba046140b658.mp4 |
| 2025-01-07 20:26:05,573 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/videos/eval/2_eval_2_a036985f900b09e578db.mp4 media/videos/eval/2_eval_2_a036985f900b09e578db.mp4 |
| 2025-01-07 20:26:05,575 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table/eval/total_success_rate_ci_6_579a029f256e694027d0.table.json media/table/eval/total_success_rate_ci_6_579a029f256e694027d0.table.json |
| 2025-01-07 20:26:05,575 INFO SenderThread:2224087 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/media/table/eval/success_rate_ci_5_8490adedbf62c026e308.table.json media/table/eval/success_rate_ci_5_8490adedbf62c026e308.table.json |
| 2025-01-07 20:26:05,575 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 20:26:05,575 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 20:26:05,576 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:05,578 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 20:26:05,578 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:05,578 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 20:26:05,578 INFO SenderThread:2224087 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 20:26:05,753 INFO wandb-upload_2:2224087 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/config.yaml |
| 2025-01-07 20:26:05,808 INFO wandb-upload_3:2224087 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/requirements.txt |
| 2025-01-07 20:26:05,808 INFO wandb-upload_0:2224087 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/wandb-summary.json |
| 2025-01-07 20:26:05,814 INFO wandb-upload_1:2224087 [upload_job.py:push():130] Uploaded file /tmp/tmp28suo57nwandb/dnrqfq78-events.out.tfevents.1736299274.leonmkim-ROG-Strix-G15CS-G15CS.2223909.0 |
| 2025-01-07 20:26:05,820 INFO wandb-upload_4:2224087 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/files/output.log |
| 2025-01-07 20:26:05,892 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 20:26:05,892 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 20:26:06,021 INFO Thread-11 (_thread_body):2224087 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 20:26:06,021 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:06,021 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 20:26:06,021 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:06,021 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 20:26:06,021 INFO SenderThread:2224087 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 20:26:06,023 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 20:26:06,023 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:06,023 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 20:26:06,023 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:06,023 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 20:26:06,023 INFO SenderThread:2224087 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 20:26:06,091 INFO SenderThread:2224087 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 20:26:06,091 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 20:26:06,091 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:06,091 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 20:26:06,091 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:06,091 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 20:26:06,091 INFO SenderThread:2224087 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 20:26:06,091 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 20:26:06,092 INFO HandlerThread:2224087 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 20:26:06,092 DEBUG SenderThread:2224087 [sender.py:send():379] send: final |
| 2025-01-07 20:26:06,092 DEBUG SenderThread:2224087 [sender.py:send():379] send: footer |
| 2025-01-07 20:26:06,092 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 20:26:06,092 INFO SenderThread:2224087 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 20:26:06,092 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 20:26:06,093 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 20:26:06,093 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 20:26:06,093 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 20:26:06,093 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 20:26:06,094 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 20:26:06,094 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 20:26:06,096 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 20:26:06,144 DEBUG SenderThread:2224087 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 20:26:06,144 INFO MainThread:2224087 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 20:26:06,144 INFO MainThread:2224087 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 20:26:06,145 INFO MainThread:2224087 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 20:26:06,145 DEBUG HandlerThread:2224087 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 20:26:06,145 INFO HandlerThread:2224087 [handler.py:finish():882] shutting down handler |
| 2025-01-07 20:26:07,094 INFO WriterThread:2224087 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1013_1/w4b02c7s/wandb/run-20250107_202114-w4b02c7s/run-w4b02c7s.wandb |
| 2025-01-07 20:26:07,144 INFO SenderThread:2224087 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 20:26:07,144 INFO SenderThread:2224087 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 20:26:07,144 INFO SenderThread:2224087 [file_pusher.py:join():175] waiting for file pusher |
|
|
