| 2025-01-04 21:09:30,841 INFO StreamThr :2022441 [internal.py:wandb_internal():85] W&B internal server running at pid: 2022441, started at: 2025-01-04 21:09:30.840683 |
| 2025-01-04 21:09:30,841 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status |
| 2025-01-04 21:09:30,853 INFO WriterThread:2022441 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/run-ot5wit9y.wandb |
| 2025-01-04 21:09:30,854 DEBUG SenderThread:2022441 [sender.py:send():379] send: header |
| 2025-01-04 21:09:30,865 DEBUG SenderThread:2022441 [sender.py:send():379] send: run |
| 2025-01-04 21:09:31,115 INFO SenderThread:2022441 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files |
| 2025-01-04 21:09:31,115 INFO SenderThread:2022441 [sender.py:_start_run_threads():1188] run started: ot5wit9y with start time 1736042970.841331 |
| 2025-01-04 21:09:31,132 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-04 21:09:31,132 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: check_version |
| 2025-01-04 21:09:31,207 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-04 21:09:31,212 DEBUG HandlerThread:2022441 [system_info.py:__init__():26] System info init |
| 2025-01-04 21:09:31,212 DEBUG HandlerThread:2022441 [system_info.py:__init__():41] System info init done |
| 2025-01-04 21:09:31,212 INFO HandlerThread:2022441 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-04 21:09:31,212 INFO SystemMonitor:2022441 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-04 21:09:31,212 INFO HandlerThread:2022441 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-04 21:09:31,213 INFO SystemMonitor:2022441 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-04 21:09:31,213 INFO SystemMonitor:2022441 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-04 21:09:31,213 INFO SystemMonitor:2022441 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-04 21:09:31,214 INFO SystemMonitor:2022441 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-04 21:09:31,215 INFO SystemMonitor:2022441 [interfaces.py:start():188] Started network monitoring |
| 2025-01-04 21:09:31,224 DEBUG HandlerThread:2022441 [system_info.py:probe():152] Probing system |
| 2025-01-04 21:09:31,226 DEBUG HandlerThread:2022441 [system_info.py:_probe_git():137] Probing git |
| 2025-01-04 21:09:31,232 DEBUG HandlerThread:2022441 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-04 21:09:31,232 DEBUG HandlerThread:2022441 [system_info.py:probe():200] Probing system done |
| 2025-01-04 21:09:31,232 DEBUG HandlerThread:2022441 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-05T02:09:31.225064', 'startedAt': '2025-01-05T02:09:30.827805', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '02cf07d901f3d8187a302cd51c9faa80321417ee'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3750.0071249999996, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.971, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 650.2632446289062}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-04 21:09:31,233 INFO HandlerThread:2022441 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-04 21:09:31,233 INFO HandlerThread:2022441 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-04 21:09:31,233 DEBUG HandlerThread:2022441 [system_info.py:_save_code():44] Saving code |
| 2025-01-04 21:09:31,234 DEBUG HandlerThread:2022441 [system_info.py:_save_code():67] Saving code done |
| 2025-01-04 21:09:31,234 DEBUG HandlerThread:2022441 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-04 21:09:31,297 DEBUG HandlerThread:2022441 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-04 21:09:31,297 INFO HandlerThread:2022441 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-04 21:09:31,300 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:09:31,300 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-04 21:09:31,301 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-04 21:09:31,301 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-04 21:09:31,535 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-04 21:09:31,535 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-04 21:09:31,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:31,537 DEBUG SenderThread:2022441 [sender.py:send():379] send: telemetry |
| 2025-01-04 21:09:31,544 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/s90pkjn7-wandb-metadata.json |
| 2025-01-04 21:09:31,571 INFO wandb-upload_2:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/fnuvzh0w-diff.patch |
| 2025-01-04 21:09:31,577 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:09:31,577 INFO HandlerThread:2022441 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/tb" |
| save: true |
| } |
| control { |
| relay_id: "0e53d899c52a40d3991c399e0b797cce" |
| } |
| _info { |
| stream_id: "ot5wit9y" |
| } |
|
|
| 2025-01-04 21:09:31,577 DEBUG HandlerThread:2022441 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-04 21:09:31,578 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:09:31,580 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/k7l7ltps-code/FISH/eval_robot.py |
| 2025-01-04 21:09:31,738 DEBUG SenderThread:2022441 [sender.py:send():379] send: tbrecord |
| 2025-01-04 21:09:31,738 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:09:31,739 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 with policy live |
| 2025-01-04 21:09:31,925 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/vieyoom2-events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/requirements.txt |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/code/FISH/eval_robot.py |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-metadata.json |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/diff.patch |
| 2025-01-04 21:09:32,117 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/code |
| 2025-01-04 21:09:32,118 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/code/FISH |
| 2025-01-04 21:09:32,535 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:33,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:34,116 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:34,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:35,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:36,535 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:09:36,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:37,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:38,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:39,409 DEBUG SenderThread:2022441 [sender.py:send():379] send: config |
| 2025-01-04 21:09:39,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:40,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:41,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:09:41,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:42,118 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:42,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:43,537 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:44,119 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:44,537 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:45,537 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:46,120 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:46,535 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:09:46,535 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:09:46,577 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:46,597 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:09:47,537 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:48,120 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:48,537 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:49,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:50,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:51,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:52,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:09:52,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:53,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:54,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:55,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:56,122 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:56,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:57,536 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:09:57,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:58,123 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:09:58,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:09:59,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:00,123 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:00,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:01,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:10:01,540 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:10:01,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:02,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:02,660 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:03,124 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/config.yaml |
| 2025-01-04 21:10:03,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:04,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:05,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:06,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:07,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:08,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:08,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:09,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:10,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:11,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:12,127 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:12,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:13,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:13,541 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:14,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:15,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:16,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:10:16,538 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:10:16,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:17,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:18,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:18,599 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:19,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:20,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:21,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:22,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:23,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:24,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:24,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:25,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:26,117 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:10:26,117 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_258e75c4654817391a5f.mp4 with policy now |
| 2025-01-04 21:10:26,117 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:10:26,118 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:10:26,118 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:10:26,119 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:10:26,130 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:26,130 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/0_eval_0_258e75c4654817391a5f.mp4 |
| 2025-01-04 21:10:26,131 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:10:26,131 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos |
| 2025-01-04 21:10:26,131 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval |
| 2025-01-04 21:10:26,131 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media |
| 2025-01-04 21:10:26,355 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/6avqu09j-media/videos/eval/0_eval_0_258e75c4654817391a5f.mp4 |
| 2025-01-04 21:10:26,544 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:26,634 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/ul4148xg-media/videos/eval/0_eval_0_258e75c4654817391a5f.mp4 |
| 2025-01-04 21:10:27,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:28,131 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:28,544 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:29,540 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:29,544 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:30,547 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:31,215 DEBUG SystemMonitor:2022441 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-04 21:10:31,216 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:10:31,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:10:31,538 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:10:31,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:32,132 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:32,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:33,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:34,133 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:34,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:35,482 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:35,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:36,134 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:10:36,372 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/wvlxpzyv-events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:10:36,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:37,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:38,134 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:38,546 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:39,546 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:40,546 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:41,075 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:41,546 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:42,136 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:42,546 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:43,546 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:44,547 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:45,547 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:46,137 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:46,218 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:46,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:10:46,539 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:10:46,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:47,547 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:48,549 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:49,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:50,557 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:51,538 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:51,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:52,139 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:52,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:53,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:54,140 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:10:54,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:55,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:56,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:10:56,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:57,552 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:58,553 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:10:59,553 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:00,142 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:00,553 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:01,219 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:11:01,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:01,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:11:01,540 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:11:01,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:02,554 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:03,555 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:04,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:05,555 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:06,555 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:06,773 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:07,555 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:08,146 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:08,240 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:11:08,240 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_e3521872760c4883ab42.mp4 with policy now |
| 2025-01-04 21:11:08,242 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:11:08,243 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:11:08,243 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:11:08,243 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:11:08,461 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/0ykbl0r1-media/videos/eval/1_eval_1_e3521872760c4883ab42.mp4 |
| 2025-01-04 21:11:08,555 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:09,146 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:11:09,147 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/1_eval_1_e3521872760c4883ab42.mp4 |
| 2025-01-04 21:11:09,147 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval |
| 2025-01-04 21:11:09,411 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/w54k1dwn-media/videos/eval/1_eval_1_e3521872760c4883ab42.mp4 |
| 2025-01-04 21:11:09,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:10,147 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:10,556 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:11,557 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:11,863 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:12,556 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:13,557 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:14,148 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:14,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:15,557 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:16,149 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:16,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:11:16,539 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:11:16,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:17,557 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:17,698 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:18,557 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:19,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:20,150 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:20,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:21,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:22,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:23,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:23,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:24,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:25,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:26,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:27,152 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:11:27,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:27,996 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:11:27,996 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:11:27,996 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:11:28,153 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:11:28,153 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:28,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:28,997 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:29,559 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:30,154 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:30,561 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:31,217 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:11:31,539 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:11:31,539 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:11:31,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:32,154 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:32,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:33,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:34,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:34,625 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:35,560 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:36,561 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:37,561 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:38,561 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:39,561 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:40,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:40,561 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:41,562 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:42,562 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:43,571 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:44,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:45,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:46,159 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:46,461 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:46,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:11:46,542 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:11:46,589 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:47,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:48,160 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:11:48,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:49,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:50,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:51,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:51,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:52,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:53,573 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:54,575 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:55,575 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:56,575 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:57,542 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:11:57,575 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:58,577 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:11:58,925 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:11:58,925 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_941ebe96bf32b79e70bb.mp4 with policy now |
| 2025-01-04 21:11:58,926 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:11:58,928 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:11:58,928 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:11:58,928 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:11:59,156 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/iato1nkl-media/videos/eval/2_eval_2_941ebe96bf32b79e70bb.mp4 |
| 2025-01-04 21:11:59,163 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:11:59,163 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/2_eval_2_941ebe96bf32b79e70bb.mp4 |
| 2025-01-04 21:11:59,163 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval |
| 2025-01-04 21:11:59,417 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/zipbhv4v-media/videos/eval/2_eval_2_941ebe96bf32b79e70bb.mp4 |
| 2025-01-04 21:11:59,581 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:00,163 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:00,580 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:01,218 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:12:01,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:12:01,545 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:12:01,585 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:02,564 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:02,605 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:03,588 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:04,165 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:04,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:05,589 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:06,165 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:06,594 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:07,599 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:07,727 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:07,822 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:12:07,822 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:12:07,832 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:12:08,166 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:12:08,166 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:12:08,398 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/bn746x97-events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:12:08,597 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:09,594 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:10,167 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:10,595 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:11,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:12,590 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:12,882 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:13,590 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:14,168 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:14,590 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:15,590 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:16,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:12:16,543 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:12:16,590 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:17,590 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:18,170 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:18,218 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:18,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:19,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:20,171 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:20,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:21,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:22,171 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:22,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:23,549 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:23,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:24,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:25,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:26,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:27,591 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:28,550 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:28,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:29,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:30,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:31,219 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:12:31,543 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:12:31,543 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:12:31,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:32,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:33,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:33,679 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:34,592 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:35,593 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:36,593 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:37,593 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:38,179 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:38,593 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:38,684 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:39,595 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:40,118 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:12:40,118 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_14a15a18ebb9ab075df2.mp4 with policy now |
| 2025-01-04 21:12:40,124 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:12:40,125 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:12:40,125 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:12:40,125 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:12:40,180 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:12:40,180 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:40,181 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/3_eval_3_14a15a18ebb9ab075df2.mp4 |
| 2025-01-04 21:12:40,181 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval |
| 2025-01-04 21:12:40,360 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/shzihwmr-media/videos/eval/3_eval_3_14a15a18ebb9ab075df2.mp4 |
| 2025-01-04 21:12:40,594 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:40,612 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/tc1mvs7u-media/videos/eval/3_eval_3_14a15a18ebb9ab075df2.mp4 |
| 2025-01-04 21:12:41,603 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:42,180 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:42,602 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:43,606 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:43,754 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:44,607 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:45,608 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:46,182 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:46,548 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:12:46,548 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:12:46,603 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:47,605 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:48,182 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:48,607 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:49,607 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:49,675 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:50,606 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:51,609 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:52,616 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:53,184 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:12:53,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:53,881 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:12:53,881 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:12:53,882 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:12:54,184 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:12:54,185 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:54,610 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:55,085 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:12:55,619 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:56,185 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:56,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:57,612 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:58,187 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:12:58,612 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:12:59,612 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:00,188 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:00,612 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:00,761 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:01,219 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:13:01,544 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:13:01,544 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:13:01,612 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:02,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:03,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:04,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:05,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:06,190 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:06,361 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:06,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:07,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:08,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:09,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:10,613 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:11,555 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:11,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:12,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:13,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:14,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:15,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:16,193 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:16,544 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:13:16,545 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:13:16,614 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:16,667 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:17,615 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:18,615 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:19,616 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:20,618 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:21,617 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:22,195 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:22,556 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:22,617 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:23,618 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:24,196 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:24,626 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:25,391 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:13:25,391 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_f55a68bbcaedd47cf64e.mp4 with policy now |
| 2025-01-04 21:13:25,401 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:13:25,401 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:13:25,401 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:25,402 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:25,614 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/2aj43i21-media/videos/eval/4_eval_4_f55a68bbcaedd47cf64e.mp4 |
| 2025-01-04 21:13:25,620 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:26,197 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:13:26,197 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:26,197 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/4_eval_4_f55a68bbcaedd47cf64e.mp4 |
| 2025-01-04 21:13:26,197 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval |
| 2025-01-04 21:13:26,470 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/zzouipef-media/videos/eval/4_eval_4_f55a68bbcaedd47cf64e.mp4 |
| 2025-01-04 21:13:26,633 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:27,572 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:27,632 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:28,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:29,628 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:30,198 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:30,632 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:31,220 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:13:31,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:13:31,559 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:13:31,639 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:32,639 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:32,917 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:33,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:34,200 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:34,639 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:35,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:36,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:37,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:38,110 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:38,201 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:13:38,443 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/f6zo02ej-events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:13:38,635 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:38,936 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:13:38,937 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:38,937 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:39,065 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-04 21:13:39,065 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-04 21:13:39,201 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:13:39,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-04 21:13:39,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media |
| 2025-01-04 21:13:39,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table/eval |
| 2025-01-04 21:13:39,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table |
| 2025-01-04 21:13:39,274 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:13:39,385 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/cpmh7585-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-04 21:13:39,635 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:39,653 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-04 21:13:39,692 INFO wandb-upload_1:2022441 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmph02m9aq9 |
| 2025-01-04 21:13:39,697 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-04 21:13:40,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:13:40,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:40,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table/eval/total_success_rate_ci_6_d64eacc3b562bec6342e.table.json |
| 2025-01-04 21:13:40,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table/eval |
| 2025-01-04 21:13:40,286 INFO SenderThread:2022441 [sender.py:send_request_log_artifact():1525] logged artifact run-ot5wit9y-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxOTE0MDYxOA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTg1MzY5NA==', 'latestArtifact': None}} |
| 2025-01-04 21:13:40,286 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:13:40,286 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now |
| 2025-01-04 21:13:40,286 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:13:40,286 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,287 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,287 DEBUG SenderThread:2022441 [sender.py:send():379] send: telemetry |
| 2025-01-04 21:13:40,287 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,291 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,291 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,291 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,292 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,292 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,292 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,292 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,293 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,293 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,293 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,293 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,294 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,294 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,294 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,294 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,295 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,295 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,295 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,295 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,296 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:40,296 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:40,296 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-04 21:13:40,466 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/gyjh6vov-media/table/eval/total_success_rate_ci_6_d64eacc3b562bec6342e.table.json |
| 2025-01-04 21:13:40,563 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/s873ui01-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-04 21:13:40,635 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:40,656 INFO wandb-upload_2:2022441 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpl783p45p |
| 2025-01-04 21:13:41,085 INFO SenderThread:2022441 [sender.py:send_request_log_artifact():1525] logged artifact run-ot5wit9y-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxOTE0MDYzMQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTg1MzY5Ng==', 'latestArtifact': None}} |
| 2025-01-04 21:13:41,085 DEBUG SenderThread:2022441 [sender.py:send():379] send: files |
| 2025-01-04 21:13:41,085 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_d64eacc3b562bec6342e.table.json with policy now |
| 2025-01-04 21:13:41,085 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:13:41,085 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:41,086 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:41,202 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:13:41,354 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/1qjb1onb-media/table/eval/total_success_rate_ci_6_d64eacc3b562bec6342e.table.json |
| 2025-01-04 21:13:41,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:42,203 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:42,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:43,558 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:43,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:44,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:45,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:46,545 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-04 21:13:46,545 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-04 21:13:46,636 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:47,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:48,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:48,642 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:49,206 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/config.yaml |
| 2025-01-04 21:13:49,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:50,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:51,637 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:51,741 DEBUG SenderThread:2022441 [sender.py:send():379] send: exit |
| 2025-01-04 21:13:51,741 INFO SenderThread:2022441 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-04 21:13:51,742 INFO SenderThread:2022441 [sender.py:send_exit():588] handling runtime: 260 |
| 2025-01-04 21:13:51,743 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:51,743 INFO SenderThread:2022441 [sender.py:send_exit():594] send defer |
| 2025-01-04 21:13:51,743 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:51,743 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-04 21:13:51,743 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:51,743 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-04 21:13:51,743 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-04 21:13:51,743 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:51,743 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-04 21:13:51,744 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:51,744 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-04 21:13:51,744 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-04 21:13:51,744 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:51,744 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-04 21:13:51,744 INFO HandlerThread:2022441 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-04 21:13:51,744 INFO HandlerThread:2022441 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-04 21:13:51,744 INFO HandlerThread:2022441 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-04 21:13:51,744 DEBUG SystemMonitor:2022441 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-04 21:13:51,744 DEBUG SystemMonitor:2022441 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-04 21:13:51,798 INFO HandlerThread:2022441 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-04 21:13:51,798 INFO HandlerThread:2022441 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-04 21:13:51,798 INFO HandlerThread:2022441 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-04 21:13:51,798 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:51,799 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-04 21:13:51,799 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-04 21:13:51,799 DEBUG SenderThread:2022441 [sender.py:send():379] send: stats |
| 2025-01-04 21:13:51,799 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:51,799 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-04 21:13:51,799 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:51,799 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-04 21:13:51,799 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-04 21:13:51,799 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:51,799 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-04 21:13:52,207 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:13:54,208 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:57,943 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 21:13:57,943 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:57,943 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:57,943 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-04 21:13:57,944 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-04 21:13:57,944 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-04 21:13:57,946 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 21:13:57,946 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:57,946 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-04 21:13:57,947 DEBUG SenderThread:2022441 [sender.py:send():379] send: history |
| 2025-01-04 21:13:57,947 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-04 21:13:57,948 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:57,948 DEBUG SenderThread:2022441 [sender.py:send():379] send: summary |
| 2025-01-04 21:13:57,952 INFO SenderThread:2022441 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-04 21:13:57,952 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:57,952 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-04 21:13:57,952 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-04 21:13:57,952 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:57,952 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-04 21:13:57,952 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:57,952 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-04 21:13:57,952 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-04 21:13:57,952 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-04 21:13:57,953 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:57,953 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-04 21:13:57,953 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:57,953 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-04 21:13:58,105 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-04 21:13:58,105 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:58,105 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-04 21:13:58,105 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:58,105 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-04 21:13:58,105 INFO SenderThread:2022441 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-04 21:13:58,108 INFO SenderThread:2022441 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-04 21:13:58,110 INFO SenderThread:2022441 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-04 21:13:58,119 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-04 21:13:58,119 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:58,119 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-04 21:13:58,120 DEBUG SenderThread:2022441 [sender.py:send():379] send: artifact |
| 2025-01-04 21:13:58,210 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:13:58,210 INFO Thread-12 :2022441 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:58,282 INFO SenderThread:2022441 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxOTEzMDY5OQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxOTEzMDY5OQ==', 'versionIndex': 419}}} |
| 2025-01-04 21:13:58,282 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:58,282 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-04 21:13:58,282 INFO SenderThread:2022441 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-04 21:13:58,743 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 21:13:59,210 INFO SenderThread:2022441 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/config.yaml config.yaml |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json wandb-summary.json |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/requirements.txt requirements.txt |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log output.log |
| 2025-01-04 21:13:59,211 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/diff.patch diff.patch |
| 2025-01-04 21:13:59,214 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-04 21:13:59,215 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/1_eval_1_e3521872760c4883ab42.mp4 media/videos/eval/1_eval_1_e3521872760c4883ab42.mp4 |
| 2025-01-04 21:13:59,215 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/2_eval_2_941ebe96bf32b79e70bb.mp4 media/videos/eval/2_eval_2_941ebe96bf32b79e70bb.mp4 |
| 2025-01-04 21:13:59,215 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/0_eval_0_258e75c4654817391a5f.mp4 media/videos/eval/0_eval_0_258e75c4654817391a5f.mp4 |
| 2025-01-04 21:13:59,215 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/4_eval_4_f55a68bbcaedd47cf64e.mp4 media/videos/eval/4_eval_4_f55a68bbcaedd47cf64e.mp4 |
| 2025-01-04 21:13:59,216 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/videos/eval/3_eval_3_14a15a18ebb9ab075df2.mp4 media/videos/eval/3_eval_3_14a15a18ebb9ab075df2.mp4 |
| 2025-01-04 21:13:59,219 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-04 21:13:59,220 INFO SenderThread:2022441 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/media/table/eval/total_success_rate_ci_6_d64eacc3b562bec6342e.table.json media/table/eval/total_success_rate_ci_6_d64eacc3b562bec6342e.table.json |
| 2025-01-04 21:13:59,220 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-04 21:13:59,220 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 21:13:59,220 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:59,220 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-04 21:13:59,220 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:59,220 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-04 21:13:59,220 INFO SenderThread:2022441 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-04 21:13:59,393 INFO wandb-upload_1:2022441 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/wandb-summary.json |
| 2025-01-04 21:13:59,401 INFO wandb-upload_2:2022441 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/config.yaml |
| 2025-01-04 21:13:59,429 INFO wandb-upload_4:2022441 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/output.log |
| 2025-01-04 21:13:59,443 INFO wandb-upload_3:2022441 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/files/requirements.txt |
| 2025-01-04 21:13:59,467 INFO wandb-upload_0:2022441 [upload_job.py:push():130] Uploaded file /tmp/tmpzhh42qunwandb/y0qhm7vn-events.out.tfevents.1736042971.leonmkim-ROG-Strix-G15CS-G15CS.2022276.0 |
| 2025-01-04 21:13:59,667 INFO Thread-11 (_thread_body):2022441 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-04 21:13:59,668 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:59,668 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-04 21:13:59,668 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:59,668 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-04 21:13:59,668 INFO SenderThread:2022441 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-04 21:13:59,669 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-04 21:13:59,669 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:59,669 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-04 21:13:59,670 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:59,670 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-04 21:13:59,670 INFO SenderThread:2022441 [file_stream.py:finish():601] file stream finish called |
| 2025-01-04 21:13:59,743 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 21:13:59,888 INFO SenderThread:2022441 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-04 21:13:59,888 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-04 21:13:59,888 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 21:13:59,889 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:59,889 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-04 21:13:59,889 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:59,889 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-04 21:13:59,889 INFO SenderThread:2022441 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-04 21:13:59,889 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-04 21:13:59,889 INFO HandlerThread:2022441 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-04 21:13:59,889 DEBUG SenderThread:2022441 [sender.py:send():379] send: final |
| 2025-01-04 21:13:59,889 DEBUG SenderThread:2022441 [sender.py:send():379] send: footer |
| 2025-01-04 21:13:59,889 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: defer |
| 2025-01-04 21:13:59,890 INFO SenderThread:2022441 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-04 21:13:59,890 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 21:13:59,890 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 21:13:59,890 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-04 21:13:59,891 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: server_info |
| 2025-01-04 21:13:59,892 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-04 21:13:59,892 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-04 21:13:59,893 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-04 21:13:59,893 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-04 21:13:59,952 DEBUG SenderThread:2022441 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-04 21:13:59,953 INFO MainThread:2022441 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-04 21:13:59,953 INFO MainThread:2022441 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-04 21:13:59,953 INFO MainThread:2022441 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-04 21:13:59,953 DEBUG HandlerThread:2022441 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-04 21:13:59,954 INFO HandlerThread:2022441 [handler.py:finish():882] shutting down handler |
| 2025-01-04 21:14:00,893 INFO WriterThread:2022441 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1922_1/ot5wit9y/wandb/run-20250104_210930-ot5wit9y/run-ot5wit9y.wandb |
| 2025-01-04 21:14:00,953 INFO SenderThread:2022441 [sender.py:finish():1615] shutting down sender |
| 2025-01-04 21:14:00,953 INFO SenderThread:2022441 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-04 21:14:00,953 INFO SenderThread:2022441 [file_pusher.py:join():175] waiting for file pusher |
|
|
