| 2024-12-31 23:44:19,877 INFO StreamThr :1769274 [internal.py:wandb_internal():85] W&B internal server running at pid: 1769274, started at: 2024-12-31 23:44:19.877147 |
| 2024-12-31 23:44:19,878 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status |
| 2024-12-31 23:44:19,889 INFO WriterThread:1769274 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/run-fkfuijgf.wandb |
| 2024-12-31 23:44:19,890 DEBUG SenderThread:1769274 [sender.py:send():379] send: header |
| 2024-12-31 23:44:19,897 DEBUG SenderThread:1769274 [sender.py:send():379] send: run |
| 2024-12-31 23:44:20,105 INFO SenderThread:1769274 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files |
| 2024-12-31 23:44:20,105 INFO SenderThread:1769274 [sender.py:_start_run_threads():1188] run started: fkfuijgf with start time 1735706659.877195 |
| 2024-12-31 23:44:20,120 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-31 23:44:20,120 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: check_version |
| 2024-12-31 23:44:20,165 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-31 23:44:20,170 DEBUG HandlerThread:1769274 [system_info.py:__init__():26] System info init |
| 2024-12-31 23:44:20,170 DEBUG HandlerThread:1769274 [system_info.py:__init__():41] System info init done |
| 2024-12-31 23:44:20,170 INFO HandlerThread:1769274 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-31 23:44:20,170 INFO SystemMonitor:1769274 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-31 23:44:20,170 INFO HandlerThread:1769274 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-31 23:44:20,171 INFO SystemMonitor:1769274 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-31 23:44:20,172 INFO SystemMonitor:1769274 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-31 23:44:20,173 INFO SystemMonitor:1769274 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-31 23:44:20,175 INFO SystemMonitor:1769274 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-31 23:44:20,175 INFO SystemMonitor:1769274 [interfaces.py:start():188] Started network monitoring |
| 2024-12-31 23:44:20,185 DEBUG HandlerThread:1769274 [system_info.py:probe():152] Probing system |
| 2024-12-31 23:44:20,187 DEBUG HandlerThread:1769274 [system_info.py:_probe_git():137] Probing git |
| 2024-12-31 23:44:20,192 DEBUG HandlerThread:1769274 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-31 23:44:20,192 DEBUG HandlerThread:1769274 [system_info.py:probe():200] Probing system done |
| 2024-12-31 23:44:20,192 DEBUG HandlerThread:1769274 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T04:44:20.186033', 'startedAt': '2025-01-01T04:44:19.867291', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3562.4945, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.02, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 609.4676246643066}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-31 23:44:20,192 INFO HandlerThread:1769274 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-31 23:44:20,192 INFO HandlerThread:1769274 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-31 23:44:20,192 DEBUG HandlerThread:1769274 [system_info.py:_save_code():44] Saving code |
| 2024-12-31 23:44:20,193 DEBUG HandlerThread:1769274 [system_info.py:_save_code():67] Saving code done |
| 2024-12-31 23:44:20,193 DEBUG HandlerThread:1769274 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-31 23:44:20,255 DEBUG HandlerThread:1769274 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-31 23:44:20,255 INFO HandlerThread:1769274 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-31 23:44:20,259 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:44:20,259 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-31 23:44:20,259 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-31 23:44:20,259 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-31 23:44:20,488 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/3et9molp-wandb-metadata.json |
| 2024-12-31 23:44:20,500 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-31 23:44:20,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-31 23:44:20,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:44:20,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:20,502 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:44:20,505 INFO HandlerThread:1769274 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/tb" |
| save: true |
| } |
| control { |
| relay_id: "8e3059a3c81d41b2beda0ecfc9f46d93" |
| } |
| _info { |
| stream_id: "fkfuijgf" |
| } |
|
|
| 2024-12-31 23:44:20,506 DEBUG HandlerThread:1769274 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-31 23:44:20,542 INFO wandb-upload_2:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/42d55c61-diff.patch |
| 2024-12-31 23:44:20,543 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/m6e1crnu-code/FISH/eval_robot.py |
| 2024-12-31 23:44:20,590 DEBUG SenderThread:1769274 [sender.py:send():379] send: telemetry |
| 2024-12-31 23:44:20,590 DEBUG SenderThread:1769274 [sender.py:send():379] send: tbrecord |
| 2024-12-31 23:44:20,590 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:44:20,590 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 with policy live |
| 2024-12-31 23:44:20,786 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/0rjdmvik-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:44:21,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:21,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/code/FISH/eval_robot.py |
| 2024-12-31 23:44:21,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:44:21,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/diff.patch |
| 2024-12-31 23:44:21,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/requirements.txt |
| 2024-12-31 23:44:21,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-metadata.json |
| 2024-12-31 23:44:21,108 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/code/FISH |
| 2024-12-31 23:44:21,108 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/code |
| 2024-12-31 23:44:21,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:22,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:23,106 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:23,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:24,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:24,662 DEBUG SenderThread:1769274 [sender.py:send():379] send: config |
| 2024-12-31 23:44:25,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:25,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:26,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:27,107 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:27,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:28,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:29,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:30,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:30,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:31,108 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:31,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:32,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:33,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:34,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:35,109 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:35,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:44:35,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:44:35,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:35,653 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:36,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:37,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:38,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:39,112 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:39,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:40,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:40,661 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:41,506 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:42,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:43,113 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:43,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:44,506 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:45,506 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:46,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:46,508 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:47,506 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:48,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:49,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:50,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:44:50,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:44:50,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:51,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:51,584 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:52,115 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/config.yaml |
| 2024-12-31 23:44:52,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:53,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:54,508 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:55,510 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:56,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:57,069 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:44:57,117 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:57,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:58,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:44:59,117 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:44:59,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:00,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:01,510 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:02,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:02,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:03,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:04,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:05,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:45:05,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:45:05,549 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:06,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:07,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:07,622 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:08,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:09,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:10,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:11,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:12,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:13,503 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:13,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:14,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:15,557 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:16,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:16,522 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:45:16,522 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_4f97d134fa70e07b1df3.mp4 with policy now |
| 2024-12-31 23:45:16,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:45:16,566 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:45:16,566 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:45:16,567 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:45:16,791 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/seuf4ki2-media/videos/eval/0_eval_0_4f97d134fa70e07b1df3.mp4 |
| 2024-12-31 23:45:17,122 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:45:17,122 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:45:17,122 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/0_eval_0_4f97d134fa70e07b1df3.mp4 |
| 2024-12-31 23:45:17,122 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media |
| 2024-12-31 23:45:17,123 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos |
| 2024-12-31 23:45:17,123 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval |
| 2024-12-31 23:45:17,423 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/1unwdjkl-media/videos/eval/0_eval_0_4f97d134fa70e07b1df3.mp4 |
| 2024-12-31 23:45:17,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:18,513 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:18,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:19,123 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:45:19,513 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:20,175 DEBUG SystemMonitor:1769274 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-31 23:45:20,176 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:45:20,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:45:20,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:45:20,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:21,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:22,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:23,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:23,619 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:24,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:25,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:26,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:27,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:28,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:29,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:29,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:30,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:31,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:32,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:33,127 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:45:33,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:34,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:34,577 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:35,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:45:35,502 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:45:35,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:36,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:37,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:38,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:39,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:39,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:40,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:41,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:42,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:43,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:44,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:45,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:45,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:46,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:47,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:48,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:49,132 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:45:49,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:50,177 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:45:50,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:45:50,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:45:50,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:50,563 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:51,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:52,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:53,133 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:45:53,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:54,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:55,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:56,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:45:56,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:57,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:58,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:45:59,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:00,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:01,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:02,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:02,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:03,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:04,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:05,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:46:05,501 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:46:05,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:06,138 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:46:06,413 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/erf1smsg-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:46:06,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:07,138 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:46:07,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:07,639 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:08,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:09,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:10,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:11,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:12,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:13,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:13,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:14,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:15,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:16,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:17,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:18,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:18,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:19,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:20,177 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:46:20,501 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:46:20,502 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:46:20,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:21,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:22,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:23,143 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:46:23,521 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:23,645 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:24,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:25,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:26,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:27,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:28,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:29,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:29,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:30,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:31,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:32,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:33,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:34,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:34,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:35,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:46:35,502 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:46:35,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:36,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:37,147 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:46:37,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:38,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:39,148 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:46:39,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:39,639 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:40,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:41,149 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:46:41,523 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:42,526 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:43,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:44,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:45,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:45,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:46,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:47,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:48,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:49,528 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:50,178 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:46:50,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:46:50,502 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:46:50,549 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:50,596 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:51,529 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:52,529 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:53,529 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:54,529 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:55,529 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:55,833 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:46:56,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:57,154 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:46:57,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:58,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:46:59,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:00,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:01,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:01,531 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:02,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:03,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:04,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:05,502 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:47:05,502 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:47:05,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:06,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:06,655 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:07,157 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:07,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:08,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:09,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:10,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:11,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:12,507 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:12,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:13,533 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:14,533 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:15,540 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:15,693 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:47:15,694 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:47:15,694 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_92c79785e0b0a9901616.mp4 with policy now |
| 2024-12-31 23:47:15,695 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:47:15,695 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:47:15,695 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:47:15,968 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/zwe35cme-media/videos/eval/1_eval_1_92c79785e0b0a9901616.mp4 |
| 2024-12-31 23:47:16,160 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:47:16,160 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/1_eval_1_92c79785e0b0a9901616.mp4 |
| 2024-12-31 23:47:16,160 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval |
| 2024-12-31 23:47:16,448 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/ac40026d-media/videos/eval/1_eval_1_92c79785e0b0a9901616.mp4 |
| 2024-12-31 23:47:16,542 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:17,160 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:17,545 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:17,697 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:18,161 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:18,546 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:19,544 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:20,178 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:47:20,505 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:47:20,505 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:47:20,549 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:21,548 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:22,550 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:23,550 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:23,587 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:24,548 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:25,553 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:26,549 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:27,554 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:28,549 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:29,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:29,555 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:30,552 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:31,554 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:32,554 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:33,165 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:33,556 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:34,556 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:34,729 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:35,511 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:47:35,511 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:47:35,555 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:36,554 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:37,555 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:38,559 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:39,560 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:40,517 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:40,557 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:41,559 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:42,560 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:43,563 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:44,562 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:45,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:45,781 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:46,563 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:47,170 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:47,564 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:48,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:49,170 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:49,605 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:50,178 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:47:50,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:47:50,505 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:47:50,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:51,171 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:47:51,564 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:51,566 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:52,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:53,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:54,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:55,565 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:56,566 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:57,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:47:57,566 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:58,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:47:59,566 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:00,174 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:48:00,432 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/3oubdk4l-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:48:00,566 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:01,047 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:48:01,048 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:48:01,048 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:48:01,174 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:01,175 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:48:01,566 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:02,569 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:03,048 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:03,175 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:03,570 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:04,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:05,176 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:05,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:48:05,505 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:48:05,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:06,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:07,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:08,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:08,582 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:09,567 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:10,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:11,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:12,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:13,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:14,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:14,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:15,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:16,569 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:17,568 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:18,569 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:19,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:19,569 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:20,179 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:48:20,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:48:20,505 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:48:20,570 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:21,570 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:22,570 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:23,570 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:24,570 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:24,620 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:25,571 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:26,571 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:27,571 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:28,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:29,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:30,515 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:30,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:31,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:32,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:33,573 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:34,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:35,504 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:48:35,505 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:48:35,572 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:35,579 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:36,573 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:37,576 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:38,405 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:48:38,405 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_942af39625588acf70b9.mp4 with policy now |
| 2024-12-31 23:48:38,408 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:48:38,408 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:48:38,409 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:48:38,409 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:48:38,574 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:38,680 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/frhyj3mk-media/videos/eval/2_eval_2_942af39625588acf70b9.mp4 |
| 2024-12-31 23:48:39,187 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:39,187 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:48:39,187 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/2_eval_2_942af39625588acf70b9.mp4 |
| 2024-12-31 23:48:39,188 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval |
| 2024-12-31 23:48:39,496 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/sh64xtaa-media/videos/eval/2_eval_2_942af39625588acf70b9.mp4 |
| 2024-12-31 23:48:39,576 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:40,187 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:40,577 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:41,188 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:41,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:41,578 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:42,577 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:43,580 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:44,579 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:45,583 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:46,584 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:47,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:47,582 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:48,581 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:49,583 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:50,179 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:48:50,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:48:50,514 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:48:50,582 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:51,583 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:52,582 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:52,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:53,584 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:54,584 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:55,584 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:56,585 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:57,587 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:57,734 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:48:58,590 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:48:59,194 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:48:59,591 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:00,587 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:01,589 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:02,594 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:03,196 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:03,518 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:03,598 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:04,599 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:05,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:49:05,510 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:49:05,601 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:06,601 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:07,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:07,908 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:49:07,909 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:49:07,909 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:49:08,197 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:49:08,198 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:49:08,444 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/ot7fldun-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:49:08,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:09,198 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:09,469 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:09,601 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:10,605 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:11,199 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:11,601 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:12,601 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:13,199 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:13,601 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:14,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:15,408 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:15,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:16,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:17,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:18,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:19,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:20,180 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:49:20,509 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:49:20,510 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:49:20,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:20,629 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:21,602 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:22,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:23,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:24,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:25,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:26,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:26,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:27,603 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:28,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:29,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:30,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:31,519 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:31,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:32,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:33,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:34,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:35,510 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:49:35,510 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:49:35,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:36,576 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:36,604 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:37,606 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:38,606 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:39,606 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:40,606 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:41,607 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:42,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:42,607 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:43,607 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:44,609 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:45,210 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:45,609 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:46,620 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:47,211 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:47,331 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:49:47,332 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_511ada0368931862d452.mp4 with policy now |
| 2024-12-31 23:49:47,335 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:49:47,335 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:49:47,335 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:49:47,357 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:49:47,605 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/r9n8aqwr-media/videos/eval/3_eval_3_511ada0368931862d452.mp4 |
| 2024-12-31 23:49:47,623 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:48,211 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:48,212 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:49:48,212 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/3_eval_3_511ada0368931862d452.mp4 |
| 2024-12-31 23:49:48,212 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval |
| 2024-12-31 23:49:48,376 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:48,568 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/p0kbzqpj-media/videos/eval/3_eval_3_511ada0368931862d452.mp4 |
| 2024-12-31 23:49:48,611 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:49,212 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:49,612 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:50,180 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:49:50,212 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:49:50,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:49:50,520 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:49:50,620 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:51,615 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:52,613 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:53,600 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:53,614 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:54,614 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:55,623 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:56,623 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:57,624 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:58,622 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:49:59,526 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:49:59,629 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:00,622 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:01,628 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:02,628 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:03,216 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:03,624 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:04,628 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:05,429 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:05,516 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:50:05,516 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:50:05,624 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:06,628 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:07,625 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:08,629 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:09,634 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:10,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:10,642 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:11,640 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:12,640 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:13,219 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:13,650 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:14,641 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:15,646 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:16,530 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:16,644 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:17,221 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:17,648 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:18,644 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:19,646 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:20,181 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:50:20,522 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:50:20,522 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:50:20,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:21,651 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:21,672 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:22,654 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:23,655 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:24,656 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:25,658 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:26,661 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:27,533 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:27,658 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:28,224 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:50:28,663 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:29,003 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:50:29,003 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:50:29,003 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:50:29,225 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:29,225 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:50:29,225 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:50:29,476 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/iq5zua9g-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:50:29,661 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:30,666 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:31,666 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:32,666 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:33,533 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:33,664 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:34,666 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:35,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:50:35,520 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:50:35,667 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:36,665 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:37,667 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:38,607 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:38,668 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:39,671 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:40,669 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:41,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:42,676 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:43,675 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:44,467 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:44,668 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:45,230 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:45,672 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:46,676 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:47,230 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:47,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:48,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:49,231 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:50:49,510 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:49,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:50,181 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:50:50,513 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:50:50,514 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:50:50,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:51,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:52,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:53,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:54,653 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:50:54,670 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:55,671 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:56,671 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:57,671 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:58,671 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:50:59,671 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:00,485 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:00,674 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:01,235 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:01,672 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:02,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:03,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:04,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:05,513 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:51:05,514 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:51:05,618 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:05,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:06,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:07,238 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:07,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:08,674 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:09,239 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:09,673 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:10,674 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:11,531 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:11,674 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:12,674 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:13,240 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:13,675 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:14,675 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:15,241 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:15,675 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:16,532 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:16,675 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:17,677 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:18,676 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:19,242 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:19,676 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:20,182 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:51:20,514 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:51:20,514 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:51:20,676 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:21,665 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:21,676 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:22,692 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:23,244 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:23,491 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:51:23,491 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_f5faf9c54c3583c3620c.mp4 with policy now |
| 2024-12-31 23:51:23,493 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:51:23,494 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:51:23,494 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:51:23,494 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:51:23,682 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:23,780 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/tfgiibxx-media/videos/eval/4_eval_4_f5faf9c54c3583c3620c.mp4 |
| 2024-12-31 23:51:24,244 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:24,244 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:51:24,245 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/4_eval_4_f5faf9c54c3583c3620c.mp4 |
| 2024-12-31 23:51:24,245 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval |
| 2024-12-31 23:51:24,554 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/3cg14ctk-media/videos/eval/4_eval_4_f5faf9c54c3583c3620c.mp4 |
| 2024-12-31 23:51:24,691 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:25,245 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:25,688 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:26,245 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:26,691 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:27,537 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:27,698 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:28,706 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:29,702 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:30,702 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:31,710 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:32,548 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:32,710 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:33,714 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:34,717 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:35,520 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:51:35,520 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:51:35,710 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:36,721 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:37,580 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:37,714 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:38,717 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:39,249 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:39,716 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:40,723 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:41,727 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:42,729 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:43,541 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:43,732 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:44,734 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:45,732 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:46,727 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:47,728 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:48,547 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:48,726 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:49,729 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:50,182 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:51:50,526 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:51:50,526 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:51:50,738 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:51,732 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:52,731 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:53,612 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:53,731 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:54,734 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:55,255 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:55,735 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:56,740 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:57,256 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:57,732 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:58,737 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:51:59,256 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:51:59,477 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:51:59,732 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:52:00,731 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:52:01,110 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:52:01,110 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:01,110 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:01,257 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:52:01,257 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:52:01,257 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:52:01,300 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 23:52:01,300 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 23:52:01,512 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:52:01,520 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/birw3f9s-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:52:01,731 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:52:01,767 INFO wandb-upload_0:1769274 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp8i279p53 |
| 2024-12-31 23:52:01,890 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 23:52:01,900 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:52:02,128 INFO SenderThread:1769274 [sender.py:send_request_log_artifact():1525] logged artifact run-fkfuijgf-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDIxNDAwNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE3OTYzMQ==', 'latestArtifact': None}} |
| 2024-12-31 23:52:02,128 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:52:02,128 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json with policy now |
| 2024-12-31 23:52:02,129 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:52:02,129 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,129 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,129 DEBUG SenderThread:1769274 [sender.py:send():379] send: telemetry |
| 2024-12-31 23:52:02,129 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,130 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,130 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,130 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,130 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,132 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,132 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,133 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,133 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,133 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,133 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,134 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,134 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,134 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,134 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,135 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,135 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,135 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,135 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,136 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,136 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:02,136 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:02,136 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/table/eval/total_success_rate_ci_6_1ee4b661c7d054c174ff.table.json |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/table/eval |
| 2024-12-31 23:52:02,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/table |
| 2024-12-31 23:52:02,328 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/y2b70qk6-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:52:02,496 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/wy349f5u-media/table/eval/total_success_rate_ci_6_1ee4b661c7d054c174ff.table.json |
| 2024-12-31 23:52:02,548 INFO wandb-upload_2:1769274 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp58yspos_ |
| 2024-12-31 23:52:02,607 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/gue7j1tz-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:52:02,731 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:52:03,040 INFO SenderThread:1769274 [sender.py:send_request_log_artifact():1525] logged artifact run-fkfuijgf-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDIxNDAxMA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE3OTYzMg==', 'latestArtifact': None}} |
| 2024-12-31 23:52:03,040 DEBUG SenderThread:1769274 [sender.py:send():379] send: files |
| 2024-12-31 23:52:03,040 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_1ee4b661c7d054c174ff.table.json with policy now |
| 2024-12-31 23:52:03,040 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:52:03,040 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:03,041 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:03,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:52:03,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:52:03,294 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/5nmobrxv-media/table/eval/total_success_rate_ci_6_1ee4b661c7d054c174ff.table.json |
| 2024-12-31 23:52:03,645 DEBUG SenderThread:1769274 [sender.py:send():379] send: exit |
| 2024-12-31 23:52:03,645 INFO SenderThread:1769274 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-31 23:52:03,646 INFO SenderThread:1769274 [sender.py:send_exit():588] handling runtime: 463 |
| 2024-12-31 23:52:03,646 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:03,646 INFO SenderThread:1769274 [sender.py:send_exit():594] send defer |
| 2024-12-31 23:52:03,646 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:03,646 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-31 23:52:03,646 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:03,646 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-31 23:52:03,646 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-31 23:52:03,647 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:03,647 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-31 23:52:03,647 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:03,647 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-31 23:52:03,647 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-31 23:52:03,647 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:03,647 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-31 23:52:03,647 INFO HandlerThread:1769274 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-31 23:52:03,647 DEBUG SystemMonitor:1769274 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-31 23:52:03,647 DEBUG SystemMonitor:1769274 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-31 23:52:03,648 INFO HandlerThread:1769274 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-31 23:52:03,648 INFO HandlerThread:1769274 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-31 23:52:03,699 INFO HandlerThread:1769274 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-31 23:52:03,699 INFO HandlerThread:1769274 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-31 23:52:03,699 INFO HandlerThread:1769274 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-31 23:52:03,700 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:03,700 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-31 23:52:03,700 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-31 23:52:03,700 DEBUG SenderThread:1769274 [sender.py:send():379] send: stats |
| 2024-12-31 23:52:03,700 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:03,700 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-31 23:52:03,700 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:03,700 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-31 23:52:03,700 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-31 23:52:03,701 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:03,701 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-31 23:52:04,258 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:52:05,259 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:52:10,115 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:52:10,116 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:52:10,116 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-31 23:52:10,116 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:52:10,117 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:10,117 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-31 23:52:10,118 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-31 23:52:10,118 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:52:10,118 DEBUG SenderThread:1769274 [sender.py:send():379] send: history |
| 2024-12-31 23:52:10,118 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:52:10,118 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:10,118 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-31 23:52:10,119 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:10,119 DEBUG SenderThread:1769274 [sender.py:send():379] send: summary |
| 2024-12-31 23:52:10,122 INFO SenderThread:1769274 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:52:10,123 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:10,123 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-31 23:52:10,123 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-31 23:52:10,123 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:10,123 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-31 23:52:10,123 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:10,123 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-31 23:52:10,126 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:52:10,251 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-31 23:52:10,251 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:10,251 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-31 23:52:10,251 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:10,251 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-31 23:52:10,261 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:52:10,261 INFO Thread-12 :1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/config.yaml |
| 2024-12-31 23:52:10,647 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:52:11,972 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-31 23:52:11,972 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:52:11,972 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:11,973 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-31 23:52:11,973 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:11,973 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-31 23:52:11,973 INFO SenderThread:1769274 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-31 23:52:11,975 INFO SenderThread:1769274 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-31 23:52:11,977 INFO SenderThread:1769274 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-31 23:52:11,990 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-31 23:52:11,991 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:11,991 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-31 23:52:11,991 DEBUG SenderThread:1769274 [sender.py:send():379] send: artifact |
| 2024-12-31 23:52:12,157 INFO SenderThread:1769274 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDIwNzU2Mg==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDIwNzU2Mg==', 'versionIndex': 401}}} |
| 2024-12-31 23:52:12,157 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:12,157 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-31 23:52:12,157 INFO SenderThread:1769274 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-31 23:52:12,262 INFO SenderThread:1769274 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:52:12,262 INFO SenderThread:1769274 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files |
| 2024-12-31 23:52:12,262 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/config.yaml config.yaml |
| 2024-12-31 23:52:12,263 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:52:12,263 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json wandb-summary.json |
| 2024-12-31 23:52:12,264 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/requirements.txt requirements.txt |
| 2024-12-31 23:52:12,265 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-31 23:52:12,265 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log output.log |
| 2024-12-31 23:52:12,265 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/diff.patch diff.patch |
| 2024-12-31 23:52:12,265 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-31 23:52:12,269 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/1_eval_1_92c79785e0b0a9901616.mp4 media/videos/eval/1_eval_1_92c79785e0b0a9901616.mp4 |
| 2024-12-31 23:52:12,270 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/4_eval_4_f5faf9c54c3583c3620c.mp4 media/videos/eval/4_eval_4_f5faf9c54c3583c3620c.mp4 |
| 2024-12-31 23:52:12,270 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/2_eval_2_942af39625588acf70b9.mp4 media/videos/eval/2_eval_2_942af39625588acf70b9.mp4 |
| 2024-12-31 23:52:12,271 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/0_eval_0_4f97d134fa70e07b1df3.mp4 media/videos/eval/0_eval_0_4f97d134fa70e07b1df3.mp4 |
| 2024-12-31 23:52:12,271 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/videos/eval/3_eval_3_511ada0368931862d452.mp4 media/videos/eval/3_eval_3_511ada0368931862d452.mp4 |
| 2024-12-31 23:52:12,271 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:52:12,271 INFO SenderThread:1769274 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/media/table/eval/total_success_rate_ci_6_1ee4b661c7d054c174ff.table.json media/table/eval/total_success_rate_ci_6_1ee4b661c7d054c174ff.table.json |
| 2024-12-31 23:52:12,271 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-31 23:52:12,271 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:12,271 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-31 23:52:12,272 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:12,272 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-31 23:52:12,272 INFO SenderThread:1769274 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 23:52:12,461 INFO wandb-upload_1:1769274 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/config.yaml |
| 2024-12-31 23:52:12,463 INFO wandb-upload_2:1769274 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/wandb-summary.json |
| 2024-12-31 23:52:12,499 INFO wandb-upload_4:1769274 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/output.log |
| 2024-12-31 23:52:12,505 INFO wandb-upload_0:1769274 [upload_job.py:push():130] Uploaded file /tmp/tmp07m76vnewandb/mdgf2ixm-events.out.tfevents.1735706660.leonmkim-ROG-Strix-G15CS-G15CS.1769082.0 |
| 2024-12-31 23:52:12,509 INFO wandb-upload_3:1769274 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/files/requirements.txt |
| 2024-12-31 23:52:12,647 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:52:12,648 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:52:12,709 INFO Thread-11 (_thread_body):1769274 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-31 23:52:12,709 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:12,709 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-31 23:52:12,709 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:12,709 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-31 23:52:12,710 INFO SenderThread:1769274 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-31 23:52:12,711 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-31 23:52:12,711 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:12,711 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-31 23:52:12,711 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:12,711 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-31 23:52:12,711 INFO SenderThread:1769274 [file_stream.py:finish():601] file stream finish called |
| 2024-12-31 23:52:12,896 INFO SenderThread:1769274 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-31 23:52:12,896 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-31 23:52:12,896 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:12,896 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-31 23:52:12,896 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:12,896 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-31 23:52:12,897 INFO SenderThread:1769274 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-31 23:52:12,897 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:52:12,897 INFO HandlerThread:1769274 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-31 23:52:12,897 DEBUG SenderThread:1769274 [sender.py:send():379] send: final |
| 2024-12-31 23:52:12,897 DEBUG SenderThread:1769274 [sender.py:send():379] send: footer |
| 2024-12-31 23:52:12,897 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:52:12,897 INFO SenderThread:1769274 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-31 23:52:12,898 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:52:12,898 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-31 23:52:12,898 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:52:12,898 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:52:12,898 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-31 23:52:12,898 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: server_info |
| 2024-12-31 23:52:12,899 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-31 23:52:12,901 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:52:12,950 DEBUG SenderThread:1769274 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:52:12,950 INFO MainThread:1769274 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-31 23:52:12,950 INFO MainThread:1769274 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-31 23:52:12,951 INFO MainThread:1769274 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-31 23:52:12,951 DEBUG HandlerThread:1769274 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-31 23:52:12,951 INFO HandlerThread:1769274 [handler.py:finish():882] shutting down handler |
| 2024-12-31 23:52:13,898 INFO WriterThread:1769274 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1745_0/fkfuijgf/wandb/run-20241231_234419-fkfuijgf/run-fkfuijgf.wandb |
| 2024-12-31 23:52:13,950 INFO SenderThread:1769274 [sender.py:finish():1615] shutting down sender |
| 2024-12-31 23:52:13,950 INFO SenderThread:1769274 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 23:52:13,950 INFO SenderThread:1769274 [file_pusher.py:join():175] waiting for file pusher |
|
|
