| 2025-01-07 17:50:46,219 INFO StreamThr :2204041 [internal.py:wandb_internal():85] W&B internal server running at pid: 2204041, started at: 2025-01-07 17:50:46.219571 |
| 2025-01-07 17:50:46,220 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 17:50:46,232 INFO WriterThread:2204041 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/run-em58uo70.wandb |
| 2025-01-07 17:50:46,232 DEBUG SenderThread:2204041 [sender.py:send():379] send: header |
| 2025-01-07 17:50:46,245 DEBUG SenderThread:2204041 [sender.py:send():379] send: run |
| 2025-01-07 17:50:46,510 INFO SenderThread:2204041 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files |
| 2025-01-07 17:50:46,510 INFO SenderThread:2204041 [sender.py:_start_run_threads():1188] run started: em58uo70 with start time 1736290246.219841 |
| 2025-01-07 17:50:46,527 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 17:50:46,527 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 17:50:46,603 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 17:50:46,608 DEBUG HandlerThread:2204041 [system_info.py:__init__():26] System info init |
| 2025-01-07 17:50:46,608 DEBUG HandlerThread:2204041 [system_info.py:__init__():41] System info init done |
| 2025-01-07 17:50:46,608 INFO HandlerThread:2204041 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 17:50:46,608 INFO SystemMonitor:2204041 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 17:50:46,608 INFO HandlerThread:2204041 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 17:50:46,609 INFO SystemMonitor:2204041 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 17:50:46,609 INFO SystemMonitor:2204041 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 17:50:46,610 INFO SystemMonitor:2204041 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 17:50:46,611 INFO SystemMonitor:2204041 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 17:50:46,611 INFO SystemMonitor:2204041 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 17:50:46,621 DEBUG HandlerThread:2204041 [system_info.py:probe():152] Probing system |
| 2025-01-07 17:50:46,622 DEBUG HandlerThread:2204041 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 17:50:46,627 DEBUG HandlerThread:2204041 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 17:50:46,627 DEBUG HandlerThread:2204041 [system_info.py:probe():200] Probing system done |
| 2025-01-07 17:50:46,627 DEBUG HandlerThread:2204041 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T22:50:46.621332', 'startedAt': '2025-01-07T22:50:46.206437', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.8066249999997, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4498.565, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.989, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 783.1284484863281}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 17:50:46,627 INFO HandlerThread:2204041 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 17:50:46,627 INFO HandlerThread:2204041 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 17:50:46,627 DEBUG HandlerThread:2204041 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 17:50:46,633 DEBUG HandlerThread:2204041 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 17:50:46,633 DEBUG HandlerThread:2204041 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 17:50:46,700 DEBUG HandlerThread:2204041 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 17:50:46,700 INFO HandlerThread:2204041 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 17:50:46,704 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:50:46,704 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 17:50:46,704 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 17:50:46,704 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 17:50:46,945 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/lchhyf4v-wandb-metadata.json |
| 2025-01-07 17:50:46,958 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 17:50:46,959 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 17:50:46,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:50:46,959 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:50:46,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:46,962 INFO HandlerThread:2204041 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/tb" |
| save: true |
| } |
| control { |
| relay_id: "9d3497cdfa604630b5ad827e26575f59" |
| } |
| _info { |
| stream_id: "em58uo70" |
| } |
|
|
| 2025-01-07 17:50:46,962 DEBUG HandlerThread:2204041 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 17:50:47,001 INFO wandb-upload_2:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/eeyox32h-diff.patch |
| 2025-01-07 17:50:47,011 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/2htrlx2o-code/FISH/eval_robot.py |
| 2025-01-07 17:50:47,070 DEBUG SenderThread:2204041 [sender.py:send():379] send: telemetry |
| 2025-01-07 17:50:47,070 DEBUG SenderThread:2204041 [sender.py:send():379] send: tbrecord |
| 2025-01-07 17:50:47,071 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:50:47,071 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 with policy live |
| 2025-01-07 17:50:47,266 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/kt5i0mta-events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:50:47,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/code/FISH/eval_robot.py |
| 2025-01-07 17:50:47,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/requirements.txt |
| 2025-01-07 17:50:47,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:50:47,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-metadata.json |
| 2025-01-07 17:50:47,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:50:47,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/diff.patch |
| 2025-01-07 17:50:47,513 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/code |
| 2025-01-07 17:50:47,513 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/code/FISH |
| 2025-01-07 17:50:47,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:48,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:49,511 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:50:49,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:50,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:51,303 DEBUG SenderThread:2204041 [sender.py:send():379] send: config |
| 2025-01-07 17:50:51,304 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:50:51,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:52,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:53,512 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:50:53,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:54,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:55,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:56,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:50:56,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:57,513 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:50:57,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:58,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:50:59,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:00,977 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:01,514 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:01,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:51:01,959 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:51:02,005 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:02,038 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:02,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:03,515 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:03,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:04,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:05,516 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:05,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:06,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:07,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:07,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:08,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:09,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:10,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:11,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:12,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:12,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:13,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:14,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:15,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:16,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:51:16,959 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:51:17,005 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:17,963 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:18,092 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:18,519 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/config.yaml |
| 2025-01-07 17:51:18,963 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:19,963 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:20,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:21,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:22,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:23,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:23,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:24,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:25,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:26,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:27,521 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:27,964 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:28,965 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:29,089 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:29,522 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:29,965 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:30,965 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:31,523 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:31,576 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:51:31,576 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_424dad725925d47367ef.mp4 with policy now |
| 2025-01-07 17:51:31,576 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:51:31,577 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:51:31,577 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:51:31,577 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:51:31,839 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/6tp6y3yi-media/videos/eval/0_eval_0_424dad725925d47367ef.mp4 |
| 2025-01-07 17:51:31,959 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:51:31,959 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:51:31,965 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:32,523 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:51:32,524 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/0_eval_0_424dad725925d47367ef.mp4 |
| 2025-01-07 17:51:32,524 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval |
| 2025-01-07 17:51:32,525 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos |
| 2025-01-07 17:51:32,525 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media |
| 2025-01-07 17:51:32,830 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/zs22w709-media/videos/eval/0_eval_0_424dad725925d47367ef.mp4 |
| 2025-01-07 17:51:32,965 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:33,524 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:33,966 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:34,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:35,055 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:35,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:36,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:37,525 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:37,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:38,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:39,526 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:39,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:40,599 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:40,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:41,527 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:51:41,527 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:41,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:42,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:43,527 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:43,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:44,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:45,528 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:45,881 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:45,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:46,612 DEBUG SystemMonitor:2204041 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 17:51:46,663 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:51:46,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:51:46,960 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:51:47,005 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:47,529 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:47,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:48,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:49,530 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:51:49,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:50,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:51,525 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:51,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:52,968 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:53,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:54,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:55,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:56,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:51:56,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:57,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:58,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:51:59,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:00,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:01,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:52:01,960 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:52:02,001 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:02,047 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:02,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:03,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:04,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:05,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:06,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:07,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:07,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:08,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:09,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:10,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:11,536 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:11,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:12,773 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:52:12,773 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_5b5bcc7391582f2bd114.mp4 with policy now |
| 2025-01-07 17:52:12,776 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:52:12,776 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:52:12,776 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:52:12,777 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:52:12,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:13,200 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/4n6po6lp-media/videos/eval/1_eval_1_5b5bcc7391582f2bd114.mp4 |
| 2025-01-07 17:52:13,537 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:13,537 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:52:13,537 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/1_eval_1_5b5bcc7391582f2bd114.mp4 |
| 2025-01-07 17:52:13,538 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval |
| 2025-01-07 17:52:13,802 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:13,828 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/na48ggjt-media/videos/eval/1_eval_1_5b5bcc7391582f2bd114.mp4 |
| 2025-01-07 17:52:13,972 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:14,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:15,538 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:15,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:16,613 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:52:16,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:52:16,962 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:52:16,977 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:17,539 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:17,979 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:18,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:19,119 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:19,979 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:20,972 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:21,540 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:21,972 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:22,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:23,348 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:52:23,349 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:52:23,349 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:52:23,541 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:52:23,541 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:52:23,806 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/f8ukn2cw-events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:52:23,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:24,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:25,101 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:25,541 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:25,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:26,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:27,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:28,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:29,543 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:29,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:30,175 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:30,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:31,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:52:31,961 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:52:32,001 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:32,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:33,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:34,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:35,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:36,057 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:36,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:37,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:38,974 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:39,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:40,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:41,967 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:41,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:42,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:43,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:44,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:45,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:46,613 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:52:46,960 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:52:46,961 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:52:47,001 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:47,090 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:47,976 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:48,976 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:49,977 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:50,977 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:51,978 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:52,969 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:52,978 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:53,978 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:54,979 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:55,081 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:52:55,081 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:52:55,082 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_2809d134000f06b2ecbc.mp4 with policy now |
| 2025-01-07 17:52:55,082 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:52:55,082 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:52:55,083 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:52:55,351 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/da1jldbk-media/videos/eval/2_eval_2_2809d134000f06b2ecbc.mp4 |
| 2025-01-07 17:52:55,551 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:55,552 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:52:55,552 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/2_eval_2_2809d134000f06b2ecbc.mp4 |
| 2025-01-07 17:52:55,552 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval |
| 2025-01-07 17:52:55,828 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/33c2ull0-media/videos/eval/2_eval_2_2809d134000f06b2ecbc.mp4 |
| 2025-01-07 17:52:55,985 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:56,985 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:57,552 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:52:57,973 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:52:57,989 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:58,982 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:52:59,987 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:00,985 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:01,553 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:01,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:53:01,961 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:53:01,984 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:02,993 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:03,119 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:03,554 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:03,986 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:04,989 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:05,183 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:53:05,183 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:53:05,184 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:53:05,555 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:53:05,555 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:05,555 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:53:05,989 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:06,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:07,555 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:07,988 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:08,700 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:08,990 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:09,556 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:09,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:10,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:11,557 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:11,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:12,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:13,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:13,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:14,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:15,996 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:16,614 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:53:16,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:53:16,961 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:53:17,001 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:17,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:18,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:19,038 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:19,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:20,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:21,560 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:21,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:22,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:23,997 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:24,970 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:24,998 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:25,998 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:26,998 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:27,998 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:28,998 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:29,971 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:29,999 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:30,999 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:31,961 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:53:31,961 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:53:32,005 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:32,999 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:33,999 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:34,999 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:35,100 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:36,003 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:36,981 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:53:36,981 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_8e5f95db56c8842b2074.mp4 with policy now |
| 2025-01-07 17:53:36,984 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:53:36,984 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:53:36,984 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:53:36,985 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:53:37,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:37,251 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/699weixh-media/videos/eval/3_eval_3_8e5f95db56c8842b2074.mp4 |
| 2025-01-07 17:53:37,566 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:37,566 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:53:37,566 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/3_eval_3_8e5f95db56c8842b2074.mp4 |
| 2025-01-07 17:53:37,566 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval |
| 2025-01-07 17:53:37,850 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/dna5w348-media/videos/eval/3_eval_3_8e5f95db56c8842b2074.mp4 |
| 2025-01-07 17:53:38,006 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:39,011 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:39,566 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:40,006 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:40,727 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:41,027 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:41,567 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:42,019 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:43,014 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:44,009 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:45,018 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:45,568 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:46,020 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:46,614 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:53:46,614 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:46,963 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:53:46,964 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:53:47,021 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:48,020 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:49,022 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:50,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:50,239 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:53:50,239 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:53:50,239 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:53:50,570 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:53:50,570 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:53:50,867 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/x5t4ko3i-events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:53:51,022 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:51,570 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:51,979 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:52,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:53,021 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:53,571 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:54,027 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:55,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:55,572 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:56,023 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:57,023 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:57,573 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:53:57,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:53:58,023 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:53:59,023 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:00,023 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:01,023 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:01,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:54:01,962 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:54:02,024 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:03,024 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:03,082 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:04,024 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:05,024 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:06,024 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:07,024 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:08,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:08,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:09,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:10,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:11,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:12,025 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:13,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:13,975 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:14,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:15,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:16,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:16,615 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:54:16,962 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:54:16,962 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:54:17,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:18,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:19,026 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:19,101 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:20,027 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:21,027 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:22,027 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:23,031 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:23,582 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:23,927 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:54:23,927 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_2b0616a340d3be9a0add.mp4 with policy now |
| 2025-01-07 17:54:23,939 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:54:23,939 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:54:23,940 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:23,940 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:24,033 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:24,186 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/rb8yjj6w-media/videos/eval/4_eval_4_2b0616a340d3be9a0add.mp4 |
| 2025-01-07 17:54:24,582 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:24,582 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/4_eval_4_2b0616a340d3be9a0add.mp4 |
| 2025-01-07 17:54:24,582 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval |
| 2025-01-07 17:54:24,913 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/7f41e2bk-media/videos/eval/4_eval_4_2b0616a340d3be9a0add.mp4 |
| 2025-01-07 17:54:24,977 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:25,037 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:25,582 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:26,040 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:27,039 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:28,041 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:29,044 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:29,584 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:30,050 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:30,520 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:31,074 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:31,584 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:31,966 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:54:31,967 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:54:32,054 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:33,066 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:34,055 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:34,308 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:54:34,309 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:34,310 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:34,541 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 17:54:34,542 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 17:54:34,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:54:34,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:34,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:54:34,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media |
| 2025-01-07 17:54:34,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table/eval |
| 2025-01-07 17:54:34,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table |
| 2025-01-07 17:54:34,727 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:54:34,784 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/wspzzxuq-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:54:35,031 INFO wandb-upload_0:2204041 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpyp8pg8x3 |
| 2025-01-07 17:54:35,055 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:35,113 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 17:54:35,157 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:54:35,453 INFO SenderThread:2204041 [sender.py:send_request_log_artifact():1525] logged artifact run-em58uo70-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjAxNzgzNw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTcyNzQ3MQ==', 'latestArtifact': None}} |
| 2025-01-07 17:54:35,453 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:54:35,453 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now |
| 2025-01-07 17:54:35,453 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:54:35,453 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,454 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,454 DEBUG SenderThread:2204041 [sender.py:send():379] send: telemetry |
| 2025-01-07 17:54:35,454 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,458 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,458 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,458 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,458 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,459 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,459 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,459 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,459 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,459 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,460 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,460 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,460 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,460 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,461 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,461 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,461 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,461 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,462 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,462 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,462 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:35,462 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:35,463 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 17:54:35,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:54:35,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:35,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:35,586 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table/eval/total_success_rate_ci_6_c48addb482dcbe6086b7.table.json |
| 2025-01-07 17:54:35,587 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table/eval |
| 2025-01-07 17:54:35,719 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/kvw6i9vs-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:54:35,774 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/uax9iuap-media/table/eval/total_success_rate_ci_6_c48addb482dcbe6086b7.table.json |
| 2025-01-07 17:54:35,845 INFO wandb-upload_2:2204041 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpx_qywsap |
| 2025-01-07 17:54:36,055 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:36,240 INFO SenderThread:2204041 [sender.py:send_request_log_artifact():1525] logged artifact run-em58uo70-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjAxNzg2NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTcyNzQ3Ng==', 'latestArtifact': None}} |
| 2025-01-07 17:54:36,245 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:36,387 DEBUG SenderThread:2204041 [sender.py:send():379] send: files |
| 2025-01-07 17:54:36,387 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_c48addb482dcbe6086b7.table.json with policy now |
| 2025-01-07 17:54:36,388 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:54:36,388 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:36,388 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:36,587 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/config.yaml |
| 2025-01-07 17:54:36,587 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:36,655 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/c4mmi5oo-media/table/eval/total_success_rate_ci_6_c48addb482dcbe6086b7.table.json |
| 2025-01-07 17:54:37,055 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:37,587 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:38,055 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:39,056 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:40,056 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:40,925 DEBUG SenderThread:2204041 [sender.py:send():379] send: exit |
| 2025-01-07 17:54:40,926 INFO SenderThread:2204041 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 17:54:40,926 INFO SenderThread:2204041 [sender.py:send_exit():588] handling runtime: 234 |
| 2025-01-07 17:54:40,926 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:40,926 INFO SenderThread:2204041 [sender.py:send_exit():594] send defer |
| 2025-01-07 17:54:40,926 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:40,926 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 17:54:40,926 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:40,927 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 17:54:40,927 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 17:54:40,927 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:40,927 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 17:54:40,927 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:40,927 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 17:54:40,927 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 17:54:40,927 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:40,927 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 17:54:40,927 INFO HandlerThread:2204041 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 17:54:40,927 DEBUG SystemMonitor:2204041 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 17:54:40,927 DEBUG SystemMonitor:2204041 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 17:54:40,928 INFO HandlerThread:2204041 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 17:54:40,928 INFO HandlerThread:2204041 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 17:54:40,997 INFO HandlerThread:2204041 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 17:54:40,997 INFO HandlerThread:2204041 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 17:54:40,997 INFO HandlerThread:2204041 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 17:54:40,998 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:40,998 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 17:54:40,998 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 17:54:40,998 DEBUG SenderThread:2204041 [sender.py:send():379] send: stats |
| 2025-01-07 17:54:40,998 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:40,998 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 17:54:40,999 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:40,999 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 17:54:40,999 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 17:54:40,999 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:40,999 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 17:54:41,589 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:43,590 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:47,315 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:54:47,315 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:47,315 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 17:54:47,315 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:47,317 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:47,317 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 17:54:47,317 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 17:54:47,317 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:54:47,317 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:47,317 DEBUG SenderThread:2204041 [sender.py:send():379] send: history |
| 2025-01-07 17:54:47,317 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 17:54:47,317 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:54:47,319 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:47,319 DEBUG SenderThread:2204041 [sender.py:send():379] send: summary |
| 2025-01-07 17:54:47,323 INFO SenderThread:2204041 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:54:47,323 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:47,323 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 17:54:47,323 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 17:54:47,323 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:47,323 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 17:54:47,324 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:47,324 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 17:54:47,324 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 17:54:47,324 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:54:47,324 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:47,324 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 17:54:47,324 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:47,324 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 17:54:47,591 INFO Thread-12 :2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:47,927 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:54:50,723 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 17:54:50,723 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:54:50,724 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:50,724 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 17:54:50,724 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:50,724 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 17:54:50,724 INFO SenderThread:2204041 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 17:54:50,726 INFO SenderThread:2204041 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 17:54:50,728 INFO SenderThread:2204041 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 17:54:50,742 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 17:54:50,743 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:50,743 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 17:54:50,743 DEBUG SenderThread:2204041 [sender.py:send():379] send: artifact |
| 2025-01-07 17:54:50,928 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:54:50,935 INFO SenderThread:2204041 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTg5NjA2OQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTkyMzk5OA==', 'versionIndex': 436}}} |
| 2025-01-07 17:54:50,936 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:50,936 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 17:54:50,936 INFO SenderThread:2204041 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 17:54:51,592 INFO SenderThread:2204041 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:51,593 INFO SenderThread:2204041 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files |
| 2025-01-07 17:54:51,593 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/config.yaml config.yaml |
| 2025-01-07 17:54:51,593 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 17:54:51,593 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/requirements.txt requirements.txt |
| 2025-01-07 17:54:51,593 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:54:51,593 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 17:54:51,594 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log output.log |
| 2025-01-07 17:54:51,594 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/diff.patch diff.patch |
| 2025-01-07 17:54:51,598 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 17:54:51,599 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/2_eval_2_2809d134000f06b2ecbc.mp4 media/videos/eval/2_eval_2_2809d134000f06b2ecbc.mp4 |
| 2025-01-07 17:54:51,599 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/4_eval_4_2b0616a340d3be9a0add.mp4 media/videos/eval/4_eval_4_2b0616a340d3be9a0add.mp4 |
| 2025-01-07 17:54:51,599 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/1_eval_1_5b5bcc7391582f2bd114.mp4 media/videos/eval/1_eval_1_5b5bcc7391582f2bd114.mp4 |
| 2025-01-07 17:54:51,599 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/3_eval_3_8e5f95db56c8842b2074.mp4 media/videos/eval/3_eval_3_8e5f95db56c8842b2074.mp4 |
| 2025-01-07 17:54:51,599 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/videos/eval/0_eval_0_424dad725925d47367ef.mp4 media/videos/eval/0_eval_0_424dad725925d47367ef.mp4 |
| 2025-01-07 17:54:51,601 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:54:51,601 INFO SenderThread:2204041 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/media/table/eval/total_success_rate_ci_6_c48addb482dcbe6086b7.table.json media/table/eval/total_success_rate_ci_6_c48addb482dcbe6086b7.table.json |
| 2025-01-07 17:54:51,601 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 17:54:51,602 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:54:51,603 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:51,603 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 17:54:51,603 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:51,603 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 17:54:51,603 INFO SenderThread:2204041 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 17:54:51,786 INFO wandb-upload_2:2204041 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/config.yaml |
| 2025-01-07 17:54:51,795 INFO wandb-upload_1:2204041 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/requirements.txt |
| 2025-01-07 17:54:51,820 INFO wandb-upload_0:2204041 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/wandb-summary.json |
| 2025-01-07 17:54:51,823 INFO wandb-upload_4:2204041 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/files/output.log |
| 2025-01-07 17:54:51,906 INFO wandb-upload_3:2204041 [upload_job.py:push():130] Uploaded file /tmp/tmplfm56fb5wandb/v587226h-events.out.tfevents.1736290246.leonmkim-ROG-Strix-G15CS-G15CS.2203862.0 |
| 2025-01-07 17:54:51,928 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:54:51,928 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:54:52,106 INFO Thread-11 (_thread_body):2204041 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 17:54:52,106 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:52,106 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 17:54:52,106 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:52,106 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 17:54:52,107 INFO SenderThread:2204041 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 17:54:52,108 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 17:54:52,108 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:52,108 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 17:54:52,108 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:52,108 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 17:54:52,108 INFO SenderThread:2204041 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 17:54:52,305 INFO SenderThread:2204041 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 17:54:52,305 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 17:54:52,305 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:52,305 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 17:54:52,305 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:52,305 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 17:54:52,306 INFO SenderThread:2204041 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 17:54:52,306 DEBUG SenderThread:2204041 [sender.py:send():379] send: final |
| 2025-01-07 17:54:52,306 DEBUG SenderThread:2204041 [sender.py:send():379] send: footer |
| 2025-01-07 17:54:52,306 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:54:52,306 INFO HandlerThread:2204041 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 17:54:52,306 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:54:52,306 INFO SenderThread:2204041 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 17:54:52,307 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:54:52,307 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 17:54:52,307 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:54:52,307 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 17:54:52,309 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:54:52,309 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 17:54:52,310 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 17:54:52,311 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:54:52,364 DEBUG SenderThread:2204041 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:54:52,365 INFO MainThread:2204041 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 17:54:52,365 INFO MainThread:2204041 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 17:54:52,365 INFO MainThread:2204041 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 17:54:52,366 DEBUG HandlerThread:2204041 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 17:54:52,366 INFO HandlerThread:2204041 [handler.py:finish():882] shutting down handler |
| 2025-01-07 17:54:53,309 INFO WriterThread:2204041 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/em58uo70/wandb/run-20250107_175046-em58uo70/run-em58uo70.wandb |
| 2025-01-07 17:54:53,364 INFO SenderThread:2204041 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 17:54:53,365 INFO SenderThread:2204041 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 17:54:53,365 INFO SenderThread:2204041 [file_pusher.py:join():175] waiting for file pusher |
|
|
