| 2025-01-07 17:14:04,431 INFO StreamThr :2196222 [internal.py:wandb_internal():85] W&B internal server running at pid: 2196222, started at: 2025-01-07 17:14:04.431363 |
| 2025-01-07 17:14:04,432 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 17:14:04,443 INFO WriterThread:2196222 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/run-6qq3p71c.wandb |
| 2025-01-07 17:14:04,444 DEBUG SenderThread:2196222 [sender.py:send():379] send: header |
| 2025-01-07 17:14:04,455 DEBUG SenderThread:2196222 [sender.py:send():379] send: run |
| 2025-01-07 17:14:04,686 INFO SenderThread:2196222 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files |
| 2025-01-07 17:14:04,687 INFO SenderThread:2196222 [sender.py:_start_run_threads():1188] run started: 6qq3p71c with start time 1736288044.431686 |
| 2025-01-07 17:14:04,702 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 17:14:04,702 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 17:14:04,776 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 17:14:04,780 DEBUG HandlerThread:2196222 [system_info.py:__init__():26] System info init |
| 2025-01-07 17:14:04,780 DEBUG HandlerThread:2196222 [system_info.py:__init__():41] System info init done |
| 2025-01-07 17:14:04,780 INFO HandlerThread:2196222 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 17:14:04,781 INFO SystemMonitor:2196222 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 17:14:04,781 INFO HandlerThread:2196222 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 17:14:04,781 INFO SystemMonitor:2196222 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 17:14:04,782 INFO SystemMonitor:2196222 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 17:14:04,783 INFO SystemMonitor:2196222 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 17:14:04,783 INFO SystemMonitor:2196222 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 17:14:04,784 INFO SystemMonitor:2196222 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 17:14:04,794 DEBUG HandlerThread:2196222 [system_info.py:probe():152] Probing system |
| 2025-01-07 17:14:04,796 DEBUG HandlerThread:2196222 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 17:14:04,800 DEBUG HandlerThread:2196222 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 17:14:04,800 DEBUG HandlerThread:2196222 [system_info.py:probe():200] Probing system done |
| 2025-01-07 17:14:04,800 DEBUG HandlerThread:2196222 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-07T22:14:04.794650', 'startedAt': '2025-01-07T22:14:04.419165', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4124.5576249999995, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4497.613, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.631, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.003, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 743.5787963867188}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 17:14:04,800 INFO HandlerThread:2196222 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 17:14:04,801 INFO HandlerThread:2196222 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 17:14:04,801 DEBUG HandlerThread:2196222 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 17:14:04,806 DEBUG HandlerThread:2196222 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 17:14:04,806 DEBUG HandlerThread:2196222 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 17:14:04,876 DEBUG HandlerThread:2196222 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 17:14:04,877 INFO HandlerThread:2196222 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 17:14:04,880 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:14:04,880 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 17:14:04,880 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 17:14:04,880 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 17:14:05,123 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 17:14:05,123 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 17:14:05,123 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:14:05,123 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:14:05,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:05,126 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/pm4k7w63-wandb-metadata.json |
| 2025-01-07 17:14:05,126 INFO HandlerThread:2196222 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/tb" |
| save: true |
| } |
| control { |
| relay_id: "38255728e4c045ed8f1dd189a41437c0" |
| } |
| _info { |
| stream_id: "6qq3p71c" |
| } |
|
|
| 2025-01-07 17:14:05,126 DEBUG HandlerThread:2196222 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 17:14:05,178 INFO wandb-upload_1:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/nl9lvu9s-code/FISH/eval_robot.py |
| 2025-01-07 17:14:05,187 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/kbip47ap-diff.patch |
| 2025-01-07 17:14:05,239 DEBUG SenderThread:2196222 [sender.py:send():379] send: telemetry |
| 2025-01-07 17:14:05,239 DEBUG SenderThread:2196222 [sender.py:send():379] send: tbrecord |
| 2025-01-07 17:14:05,240 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:14:05,240 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 with policy live |
| 2025-01-07 17:14:05,432 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/crfyvop6-events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:14:05,688 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/code/FISH/eval_robot.py |
| 2025-01-07 17:14:05,688 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-metadata.json |
| 2025-01-07 17:14:05,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/requirements.txt |
| 2025-01-07 17:14:05,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:14:05,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/diff.patch |
| 2025-01-07 17:14:05,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:05,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/code/FISH |
| 2025-01-07 17:14:05,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/code |
| 2025-01-07 17:14:06,123 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:07,123 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:07,445 DEBUG SenderThread:2196222 [sender.py:send():379] send: config |
| 2025-01-07 17:14:07,688 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:08,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:09,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:09,497 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:09,688 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:10,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:11,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:12,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:13,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:13,689 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:14,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:14,885 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:15,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:16,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:17,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:17,690 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:18,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:19,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:20,123 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:14:20,123 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:14:20,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:20,250 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:21,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:21,692 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:22,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:23,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:23,692 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:24,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:25,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:25,693 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:26,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:26,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:27,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:28,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:29,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:30,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:31,124 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:31,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:32,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:33,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:34,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:35,123 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:14:35,123 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:14:35,165 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:36,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:36,225 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:36,696 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/config.yaml |
| 2025-01-07 17:14:37,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:38,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:39,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:40,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:41,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:42,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:42,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:43,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:44,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:45,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:46,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:47,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:48,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:48,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:49,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:50,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:14:50,125 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:14:50,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:50,423 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:14:50,423 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_393b812fc042bd7e1762.mp4 with policy now |
| 2025-01-07 17:14:50,423 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:14:50,424 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:14:50,424 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:14:50,426 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:14:50,673 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/wmm0r09y-media/videos/eval/0_eval_0_393b812fc042bd7e1762.mp4 |
| 2025-01-07 17:14:50,699 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:14:50,700 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/0_eval_0_393b812fc042bd7e1762.mp4 |
| 2025-01-07 17:14:50,700 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media |
| 2025-01-07 17:14:50,700 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval |
| 2025-01-07 17:14:50,701 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos |
| 2025-01-07 17:14:50,968 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/n6l1p8xt-media/videos/eval/0_eval_0_393b812fc042bd7e1762.mp4 |
| 2025-01-07 17:14:51,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:51,700 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:52,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:53,130 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:53,430 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:54,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:55,130 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:55,701 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:56,130 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:57,130 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:57,702 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:14:58,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:14:58,681 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:14:59,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:00,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:01,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:01,703 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:15:02,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:03,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:03,704 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:04,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:04,593 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:04,784 DEBUG SystemMonitor:2196222 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 17:15:04,785 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:15:05,127 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:15:05,127 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:15:05,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:06,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:07,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:07,705 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:08,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:09,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:09,861 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:10,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:11,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:11,706 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:12,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:13,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:14,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:15,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:15,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:16,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:17,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:18,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:19,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:20,125 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:15:20,125 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:15:20,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:20,239 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:21,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:22,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:23,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:24,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:25,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:25,711 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:26,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:26,134 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:27,134 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:28,134 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:29,134 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:30,135 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:31,136 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:32,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:32,136 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:33,136 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:33,445 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:15:33,445 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:15:33,445 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_19268ebe421ff04fe7f0.mp4 with policy now |
| 2025-01-07 17:15:33,446 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:15:33,446 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:15:33,447 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:15:33,706 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/zg8yj1wa-media/videos/eval/1_eval_1_19268ebe421ff04fe7f0.mp4 |
| 2025-01-07 17:15:33,713 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:33,714 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:15:33,714 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/1_eval_1_19268ebe421ff04fe7f0.mp4 |
| 2025-01-07 17:15:33,714 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval |
| 2025-01-07 17:15:33,980 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/3n0ami7s-media/videos/eval/1_eval_1_19268ebe421ff04fe7f0.mp4 |
| 2025-01-07 17:15:34,138 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:34,785 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:15:35,130 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:15:35,130 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:15:35,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:35,714 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:36,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:37,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:37,993 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:38,144 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:39,143 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:39,715 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:40,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:41,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:41,716 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:42,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:43,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:43,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:44,140 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:44,518 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:15:44,518 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:15:44,519 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:15:44,717 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:15:44,717 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:15:44,978 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/pzupy8xk-events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:15:45,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:45,718 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:46,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:47,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:47,718 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:48,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:48,953 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:49,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:49,719 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:15:50,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:15:50,126 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:15:50,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:51,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:52,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:53,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:54,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:54,293 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:15:55,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:56,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:57,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:58,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:15:59,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:00,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:00,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:01,142 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:02,144 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:03,143 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:04,144 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:04,786 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:16:05,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:16:05,126 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:16:05,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:05,222 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:05,725 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:06,144 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:07,145 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:08,145 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:09,145 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:10,145 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:11,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:11,145 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:12,145 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:13,146 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:14,146 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:15,149 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:15,728 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:15,950 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:16:15,950 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_d676baf746a2d4745eff.mp4 with policy now |
| 2025-01-07 17:16:15,958 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:16:15,958 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:16:15,958 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:16:15,959 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:16:16,150 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:16,202 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/4er4aazk-media/videos/eval/2_eval_2_d676baf746a2d4745eff.mp4 |
| 2025-01-07 17:16:16,728 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:16:16,728 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/2_eval_2_d676baf746a2d4745eff.mp4 |
| 2025-01-07 17:16:16,728 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval |
| 2025-01-07 17:16:16,982 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:17,009 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/3i7cr54k-media/videos/eval/2_eval_2_d676baf746a2d4745eff.mp4 |
| 2025-01-07 17:16:17,151 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:17,729 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:18,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:18,729 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:19,150 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:20,129 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:16:20,129 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:16:20,152 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:21,154 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:21,731 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:22,152 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:22,272 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:23,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:23,731 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:24,156 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:25,155 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:26,157 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:27,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:28,131 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:28,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:29,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:29,733 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:16:30,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:30,354 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:16:30,354 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:16:30,355 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:16:30,734 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:16:31,158 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:31,734 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:32,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:33,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:33,396 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:34,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:34,786 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:16:35,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:16:35,126 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:16:35,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:35,735 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:16:36,158 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:37,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:38,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:39,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:39,215 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:40,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:41,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:42,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:43,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:44,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:45,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:45,159 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:46,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:47,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:48,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:49,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:50,126 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:16:50,126 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:16:50,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:50,291 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:51,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:52,160 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:53,161 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:54,161 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:55,161 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:56,132 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:16:56,161 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:57,161 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:58,161 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:16:59,164 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:00,164 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:01,165 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:01,317 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:01,661 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:17:01,661 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_0c26e92b3a127f92ec68.mp4 with policy now |
| 2025-01-07 17:17:01,662 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:17:01,663 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:17:01,663 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:01,663 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:01,744 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:01,744 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:17:01,744 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/3_eval_3_0c26e92b3a127f92ec68.mp4 |
| 2025-01-07 17:17:01,745 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval |
| 2025-01-07 17:17:01,922 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/qf3aacrc-media/videos/eval/3_eval_3_0c26e92b3a127f92ec68.mp4 |
| 2025-01-07 17:17:02,164 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:02,189 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/pjenyrtw-media/videos/eval/3_eval_3_0c26e92b3a127f92ec68.mp4 |
| 2025-01-07 17:17:03,165 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:03,745 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:04,167 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:04,787 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:17:05,138 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:17:05,138 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:17:05,185 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:06,167 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:07,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:07,259 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:07,746 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:08,167 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:09,168 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:09,747 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:09,747 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:17:10,000 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/d6zk95u6-events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:17:10,168 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:10,406 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:17:10,406 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:10,406 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:10,747 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:17:11,171 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:11,748 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:12,171 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:12,585 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:13,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:14,168 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:15,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:15,749 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:16,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:17,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:18,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:18,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:19,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:20,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:17:20,128 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:17:20,169 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:21,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:22,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:23,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:23,224 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:24,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:25,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:26,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:27,170 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:28,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:29,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:29,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:30,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:31,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:32,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:33,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:34,133 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:34,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:34,787 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:17:35,128 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:17:35,128 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:17:35,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:36,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:37,172 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:38,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:39,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:39,222 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:40,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:41,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:41,758 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:41,822 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:17:41,822 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_d693180a6d9e048e7c1d.mp4 with policy now |
| 2025-01-07 17:17:41,827 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:17:41,828 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:17:41,828 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:41,828 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:42,078 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/32uoeb2q-media/videos/eval/4_eval_4_d693180a6d9e048e7c1d.mp4 |
| 2025-01-07 17:17:42,173 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:42,758 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:17:42,758 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/4_eval_4_d693180a6d9e048e7c1d.mp4 |
| 2025-01-07 17:17:42,758 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval |
| 2025-01-07 17:17:43,070 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/s3n9j6oy-media/videos/eval/4_eval_4_d693180a6d9e048e7c1d.mp4 |
| 2025-01-07 17:17:43,175 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:43,758 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:44,181 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:44,759 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:44,852 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:45,180 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:46,178 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:47,182 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:47,760 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:48,186 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:49,186 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:49,761 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:50,144 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:50,146 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 17:17:50,146 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 17:17:50,190 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:51,185 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:51,762 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:17:52,081 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 17:17:52,082 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 17:17:52,183 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:52,262 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:17:52,601 INFO wandb-upload_2:2196222 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp_hat441t |
| 2025-01-07 17:17:52,635 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 17:17:52,643 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 17:17:52,762 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:17:52,762 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:17:52,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/table/eval/total_success_rate_ci_6_32f8ff8bc7debb3475b0.table.json |
| 2025-01-07 17:17:52,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media |
| 2025-01-07 17:17:52,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/table/eval |
| 2025-01-07 17:17:52,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/table |
| 2025-01-07 17:17:53,001 INFO SenderThread:2196222 [sender.py:send_request_log_artifact():1525] logged artifact run-6qq3p71c-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTkwOTQ1Ng==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTY5NjEwOQ==', 'latestArtifact': None}} |
| 2025-01-07 17:17:53,001 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:17:53,001 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json with policy now |
| 2025-01-07 17:17:53,001 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:17:53,001 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,001 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,002 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:17:53,002 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,002 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,003 DEBUG SenderThread:2196222 [sender.py:send():379] send: telemetry |
| 2025-01-07 17:17:53,003 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,003 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,003 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,003 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,004 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,004 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,004 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,004 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,005 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,005 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,005 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,005 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,005 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,006 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,006 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,006 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,006 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,007 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,007 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,007 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,007 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,007 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:53,008 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 17:17:53,177 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/6c7xiqr1-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:17:53,181 INFO wandb-upload_1:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/wq18ni64-media/table/eval/total_success_rate_ci_6_32f8ff8bc7debb3475b0.table.json |
| 2025-01-07 17:17:53,183 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:53,417 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/p7744o24-media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:17:53,482 INFO wandb-upload_2:2196222 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppgxspz31 |
| 2025-01-07 17:17:53,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:17:53,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:17:53,842 INFO SenderThread:2196222 [sender.py:send_request_log_artifact():1525] logged artifact run-6qq3p71c-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNTkwOTQ5MA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTY5NjExMQ==', 'latestArtifact': None}} |
| 2025-01-07 17:17:53,842 DEBUG SenderThread:2196222 [sender.py:send():379] send: files |
| 2025-01-07 17:17:53,842 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_32f8ff8bc7debb3475b0.table.json with policy now |
| 2025-01-07 17:17:53,843 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:17:53,843 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:17:53,843 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:54,078 INFO wandb-upload_1:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/hyz89z7w-media/table/eval/total_success_rate_ci_6_32f8ff8bc7debb3475b0.table.json |
| 2025-01-07 17:17:54,184 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:54,763 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:17:55,184 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:56,141 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:17:56,185 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:56,764 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/config.yaml |
| 2025-01-07 17:17:57,185 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:58,185 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:17:58,686 DEBUG SenderThread:2196222 [sender.py:send():379] send: exit |
| 2025-01-07 17:17:58,686 INFO SenderThread:2196222 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 17:17:58,686 INFO SenderThread:2196222 [sender.py:send_exit():588] handling runtime: 233 |
| 2025-01-07 17:17:58,686 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:17:58,686 INFO SenderThread:2196222 [sender.py:send_exit():594] send defer |
| 2025-01-07 17:17:58,686 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:17:58,686 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 17:17:58,687 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:17:58,687 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 17:17:58,687 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 17:17:58,687 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:17:58,687 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 17:17:58,687 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:17:58,687 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 17:17:58,687 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 17:17:58,687 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:17:58,687 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 17:17:58,688 INFO HandlerThread:2196222 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 17:17:58,688 DEBUG SystemMonitor:2196222 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 17:17:58,688 DEBUG SystemMonitor:2196222 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 17:17:58,688 INFO HandlerThread:2196222 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 17:17:58,689 INFO HandlerThread:2196222 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 17:17:58,740 INFO HandlerThread:2196222 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 17:17:58,741 INFO HandlerThread:2196222 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 17:17:58,741 INFO HandlerThread:2196222 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 17:17:58,741 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:17:58,741 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 17:17:58,741 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 17:17:58,741 DEBUG SenderThread:2196222 [sender.py:send():379] send: stats |
| 2025-01-07 17:17:58,742 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:17:58,742 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 17:17:58,742 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:17:58,742 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 17:17:58,742 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 17:17:58,742 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:17:58,742 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 17:17:58,765 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:17:59,765 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:18:05,460 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:18:05,461 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:18:05,461 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 17:18:05,462 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:05,462 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 17:18:05,462 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 17:18:05,462 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:05,462 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:18:05,462 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 17:18:05,462 DEBUG SenderThread:2196222 [sender.py:send():379] send: history |
| 2025-01-07 17:18:05,463 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 17:18:05,464 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:18:05,464 DEBUG SenderThread:2196222 [sender.py:send():379] send: summary |
| 2025-01-07 17:18:05,468 INFO SenderThread:2196222 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 17:18:05,468 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:05,468 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 17:18:05,468 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 17:18:05,468 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:05,468 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 17:18:05,469 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:05,469 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 17:18:05,469 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 17:18:05,469 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 17:18:05,469 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:05,469 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 17:18:05,469 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:05,469 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 17:18:05,687 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:18:05,768 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:18:05,928 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 17:18:05,928 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:18:05,928 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:05,928 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 17:18:05,928 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:05,928 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 17:18:05,928 INFO SenderThread:2196222 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 17:18:05,931 INFO SenderThread:2196222 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 17:18:05,933 INFO SenderThread:2196222 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 17:18:05,946 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 17:18:05,946 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:05,946 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 17:18:05,946 DEBUG SenderThread:2196222 [sender.py:send():379] send: artifact |
| 2025-01-07 17:18:06,151 INFO SenderThread:2196222 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNTg5NjA2OQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNTg5NjA2OQ==', 'versionIndex': 435}}} |
| 2025-01-07 17:18:06,151 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:06,151 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 17:18:06,151 INFO SenderThread:2196222 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 17:18:06,687 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:18:06,768 INFO Thread-12 :2196222 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/config.yaml config.yaml |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/requirements.txt requirements.txt |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log output.log |
| 2025-01-07 17:18:06,769 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/diff.patch diff.patch |
| 2025-01-07 17:18:06,774 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 17:18:06,776 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/2_eval_2_d676baf746a2d4745eff.mp4 media/videos/eval/2_eval_2_d676baf746a2d4745eff.mp4 |
| 2025-01-07 17:18:06,776 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/3_eval_3_0c26e92b3a127f92ec68.mp4 media/videos/eval/3_eval_3_0c26e92b3a127f92ec68.mp4 |
| 2025-01-07 17:18:06,776 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/1_eval_1_19268ebe421ff04fe7f0.mp4 media/videos/eval/1_eval_1_19268ebe421ff04fe7f0.mp4 |
| 2025-01-07 17:18:06,776 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/4_eval_4_d693180a6d9e048e7c1d.mp4 media/videos/eval/4_eval_4_d693180a6d9e048e7c1d.mp4 |
| 2025-01-07 17:18:06,777 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/videos/eval/0_eval_0_393b812fc042bd7e1762.mp4 media/videos/eval/0_eval_0_393b812fc042bd7e1762.mp4 |
| 2025-01-07 17:18:06,778 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/table/eval/total_success_rate_ci_6_32f8ff8bc7debb3475b0.table.json media/table/eval/total_success_rate_ci_6_32f8ff8bc7debb3475b0.table.json |
| 2025-01-07 17:18:06,778 INFO SenderThread:2196222 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json media/table/eval/success_rate_ci_5_d8821c50291f0fbf979f.table.json |
| 2025-01-07 17:18:06,778 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 17:18:06,778 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:18:06,778 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:06,779 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 17:18:06,779 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:06,779 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 17:18:06,779 INFO SenderThread:2196222 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 17:18:06,951 INFO wandb-upload_1:2196222 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/requirements.txt |
| 2025-01-07 17:18:06,952 INFO wandb-upload_2:2196222 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/config.yaml |
| 2025-01-07 17:18:06,984 INFO wandb-upload_4:2196222 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/output.log |
| 2025-01-07 17:18:07,000 INFO wandb-upload_0:2196222 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/files/wandb-summary.json |
| 2025-01-07 17:18:07,045 INFO wandb-upload_3:2196222 [upload_job.py:push():130] Uploaded file /tmp/tmpqcd60463wandb/urz358fq-events.out.tfevents.1736288045.leonmkim-ROG-Strix-G15CS-G15CS.2196057.0 |
| 2025-01-07 17:18:07,246 INFO Thread-11 (_thread_body):2196222 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 17:18:07,246 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:07,246 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 17:18:07,247 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:07,247 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 17:18:07,247 INFO SenderThread:2196222 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 17:18:07,248 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 17:18:07,248 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:07,248 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 17:18:07,248 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:07,248 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 17:18:07,248 INFO SenderThread:2196222 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 17:18:07,538 INFO SenderThread:2196222 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 17:18:07,538 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 17:18:07,538 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:07,538 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 17:18:07,538 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:07,538 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 17:18:07,539 INFO SenderThread:2196222 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 17:18:07,539 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 17:18:07,539 INFO HandlerThread:2196222 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 17:18:07,539 DEBUG SenderThread:2196222 [sender.py:send():379] send: final |
| 2025-01-07 17:18:07,539 DEBUG SenderThread:2196222 [sender.py:send():379] send: footer |
| 2025-01-07 17:18:07,539 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 17:18:07,539 INFO SenderThread:2196222 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 17:18:07,539 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:18:07,540 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:18:07,540 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 17:18:07,540 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 17:18:07,543 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 17:18:07,543 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 17:18:07,544 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 17:18:07,545 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 17:18:07,588 DEBUG SenderThread:2196222 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 17:18:07,589 INFO MainThread:2196222 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 17:18:07,589 INFO MainThread:2196222 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 17:18:07,589 INFO MainThread:2196222 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 17:18:07,589 DEBUG HandlerThread:2196222 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 17:18:07,589 INFO HandlerThread:2196222 [handler.py:finish():882] shutting down handler |
| 2025-01-07 17:18:08,544 INFO WriterThread:2196222 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1009_1/6qq3p71c/wandb/run-20250107_171404-6qq3p71c/run-6qq3p71c.wandb |
| 2025-01-07 17:18:08,589 INFO SenderThread:2196222 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 17:18:08,589 INFO SenderThread:2196222 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 17:18:08,589 INFO SenderThread:2196222 [file_pusher.py:join():175] waiting for file pusher |
|
|
