| 2025-01-07 21:49:39,046 INFO StreamThr :2237540 [internal.py:wandb_internal():85] W&B internal server running at pid: 2237540, started at: 2025-01-07 21:49:39.046048 |
| 2025-01-07 21:49:39,047 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status |
| 2025-01-07 21:49:39,061 INFO WriterThread:2237540 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/run-7u1dchof.wandb |
| 2025-01-07 21:49:39,062 DEBUG SenderThread:2237540 [sender.py:send():379] send: header |
| 2025-01-07 21:49:39,069 DEBUG SenderThread:2237540 [sender.py:send():379] send: run |
| 2025-01-07 21:49:39,302 INFO SenderThread:2237540 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files |
| 2025-01-07 21:49:39,302 INFO SenderThread:2237540 [sender.py:_start_run_threads():1188] run started: 7u1dchof with start time 1736304579.046338 |
| 2025-01-07 21:49:39,316 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-07 21:49:39,316 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: check_version |
| 2025-01-07 21:49:39,361 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-07 21:49:39,366 DEBUG HandlerThread:2237540 [system_info.py:__init__():26] System info init |
| 2025-01-07 21:49:39,366 DEBUG HandlerThread:2237540 [system_info.py:__init__():41] System info init done |
| 2025-01-07 21:49:39,366 INFO HandlerThread:2237540 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-07 21:49:39,366 INFO SystemMonitor:2237540 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-07 21:49:39,366 INFO SystemMonitor:2237540 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-07 21:49:39,367 INFO SystemMonitor:2237540 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-07 21:49:39,367 INFO HandlerThread:2237540 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-07 21:49:39,367 INFO SystemMonitor:2237540 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-07 21:49:39,368 INFO SystemMonitor:2237540 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-07 21:49:39,368 INFO SystemMonitor:2237540 [interfaces.py:start():188] Started network monitoring |
| 2025-01-07 21:49:39,378 DEBUG HandlerThread:2237540 [system_info.py:probe():152] Probing system |
| 2025-01-07 21:49:39,380 DEBUG HandlerThread:2237540 [system_info.py:_probe_git():137] Probing git |
| 2025-01-07 21:49:39,384 DEBUG HandlerThread:2237540 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-07 21:49:39,384 DEBUG HandlerThread:2237540 [system_info.py:probe():200] Probing system done |
| 2025-01-07 21:49:39,384 DEBUG HandlerThread:2237540 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-08T02:49:39.378676', 'startedAt': '2025-01-08T02:49:39.039301', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '29237eae507484d4aff7683bad40a564046255b9'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3558.6397500000003, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4485.477, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.04, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4498.251, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 637.9640197753906}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-07 21:49:39,384 INFO HandlerThread:2237540 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-07 21:49:39,384 INFO HandlerThread:2237540 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-07 21:49:39,384 DEBUG HandlerThread:2237540 [system_info.py:_save_code():44] Saving code |
| 2025-01-07 21:49:39,386 DEBUG HandlerThread:2237540 [system_info.py:_save_code():67] Saving code done |
| 2025-01-07 21:49:39,386 DEBUG HandlerThread:2237540 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-07 21:49:39,451 DEBUG HandlerThread:2237540 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-07 21:49:39,451 INFO HandlerThread:2237540 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-07 21:49:39,454 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:49:39,454 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-07 21:49:39,455 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-07 21:49:39,455 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-07 21:49:39,693 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-07 21:49:39,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:49:39,694 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-07 21:49:39,694 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/ohqyrws2-wandb-metadata.json |
| 2025-01-07 21:49:39,695 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:49:39,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:39,697 INFO HandlerThread:2237540 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/tb" |
| save: true |
| } |
| control { |
| relay_id: "b280c0d6940a4957b38b21bf92a79360" |
| } |
| _info { |
| stream_id: "7u1dchof" |
| } |
|
|
| 2025-01-07 21:49:39,697 DEBUG HandlerThread:2237540 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-07 21:49:39,705 INFO wandb-upload_2:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/k05q5dgz-diff.patch |
| 2025-01-07 21:49:39,760 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/ukefgs7x-code/FISH/eval_robot.py |
| 2025-01-07 21:49:39,797 DEBUG SenderThread:2237540 [sender.py:send():379] send: telemetry |
| 2025-01-07 21:49:39,798 DEBUG SenderThread:2237540 [sender.py:send():379] send: tbrecord |
| 2025-01-07 21:49:39,798 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:49:39,798 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 with policy live |
| 2025-01-07 21:49:40,017 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/lr08xa1p-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:49:40,303 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/requirements.txt |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/code/FISH/eval_robot.py |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/diff.patch |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-metadata.json |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/code |
| 2025-01-07 21:49:40,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/code/FISH |
| 2025-01-07 21:49:40,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:41,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:42,303 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:42,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:43,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:44,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:49:44,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:45,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:46,481 DEBUG SenderThread:2237540 [sender.py:send():379] send: config |
| 2025-01-07 21:49:46,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:47,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:48,304 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:48,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:49,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:49:49,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:50,305 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:50,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:51,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:52,306 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:52,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:53,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:54,693 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:49:54,694 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:49:54,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:54,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:49:55,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:56,307 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:56,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:57,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:58,307 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:49:58,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:49:59,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:00,165 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:00,308 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:00,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:01,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:02,309 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:02,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:03,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:04,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:05,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:05,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:06,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:07,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:08,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:09,693 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:50:09,694 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:50:09,741 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:10,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:10,810 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:11,311 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/config.yaml |
| 2025-01-07 21:50:11,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:12,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:13,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:14,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:15,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:16,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:16,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:17,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:18,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:19,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:20,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:21,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:21,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:22,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:23,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:24,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:50:24,694 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:50:24,737 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:25,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:26,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:26,773 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:27,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:28,162 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:50:28,162 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_165af090a1eeabfb388b.mp4 with policy now |
| 2025-01-07 21:50:28,162 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:50:28,163 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:50:28,163 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:50:28,163 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:50:28,315 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:28,316 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/0_eval_0_165af090a1eeabfb388b.mp4 |
| 2025-01-07 21:50:28,316 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:50:28,316 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media |
| 2025-01-07 21:50:28,316 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:50:28,316 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos |
| 2025-01-07 21:50:28,461 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/lwao7e3p-media/videos/eval/0_eval_0_165af090a1eeabfb388b.mp4 |
| 2025-01-07 21:50:28,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:28,742 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/r1mxz7xl-media/videos/eval/0_eval_0_165af090a1eeabfb388b.mp4 |
| 2025-01-07 21:50:29,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:30,316 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:30,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:31,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:32,621 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:32,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:33,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:34,317 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:34,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:35,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:36,318 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:36,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:37,696 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:37,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:38,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:39,368 DEBUG SystemMonitor:2237540 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-07 21:50:39,369 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:50:39,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:50:39,694 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:50:39,737 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:40,319 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:50:40,588 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/pvxw69ed-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:50:40,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:41,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:42,320 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:42,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:43,165 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:43,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:44,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:45,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:46,321 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:50:46,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:47,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:48,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:48,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:49,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:50,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:51,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:52,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:53,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:53,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:54,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:50:54,695 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:50:54,741 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:55,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:56,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:57,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:58,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:50:58,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:50:59,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:00,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:01,705 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:02,326 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:02,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:03,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:03,858 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:04,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:05,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:06,328 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:06,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:07,707 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:08,328 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:08,707 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:09,253 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:09,370 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:51:09,694 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:51:09,695 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:51:09,741 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:10,707 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:11,708 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:12,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:51:12,085 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:51:12,086 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_a4095f09212ed4cd67c6.mp4 with policy now |
| 2025-01-07 21:51:12,086 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:51:12,086 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:51:12,086 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:51:12,330 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:12,330 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:51:12,330 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/1_eval_1_a4095f09212ed4cd67c6.mp4 |
| 2025-01-07 21:51:12,330 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:51:12,347 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/7w50oiu3-media/videos/eval/1_eval_1_a4095f09212ed4cd67c6.mp4 |
| 2025-01-07 21:51:12,615 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/smxnqzhc-media/videos/eval/1_eval_1_a4095f09212ed4cd67c6.mp4 |
| 2025-01-07 21:51:12,709 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:13,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:14,330 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:14,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:14,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:15,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:16,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:17,715 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:18,332 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:18,712 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:19,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:19,712 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:20,333 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:20,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:21,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:22,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:23,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:24,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:51:24,695 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:51:24,737 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:24,818 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:25,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:26,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:27,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:28,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:29,336 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:51:29,713 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:30,125 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:51:30,126 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:51:30,126 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:30,126 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:51:30,336 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:30,337 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:51:30,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:31,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:32,337 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:32,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:33,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:34,338 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:51:34,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:35,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:35,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:36,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:37,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:38,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:39,370 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:51:39,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:51:39,695 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:51:39,737 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:40,715 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:40,816 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:41,715 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:42,715 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:43,715 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:44,717 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:45,717 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:46,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:46,717 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:47,718 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:48,718 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:49,718 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:50,718 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:51,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:52,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:52,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:53,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:54,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:51:54,695 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:51:54,737 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:55,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:56,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:57,720 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:57,834 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:51:58,720 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:51:59,720 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:00,725 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:01,545 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:52:01,545 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_f1a3689635b40f332568.mp4 with policy now |
| 2025-01-07 21:52:01,549 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:52:01,549 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:52:01,550 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:52:01,551 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:52:01,721 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:01,810 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/u6icwa1c-media/videos/eval/2_eval_2_f1a3689635b40f332568.mp4 |
| 2025-01-07 21:52:02,348 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:02,348 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:52:02,348 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/2_eval_2_f1a3689635b40f332568.mp4 |
| 2025-01-07 21:52:02,348 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:52:02,660 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/bwtpnajq-media/videos/eval/2_eval_2_f1a3689635b40f332568.mp4 |
| 2025-01-07 21:52:02,726 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:03,348 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:03,573 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:03,726 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:04,348 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:04,728 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:05,730 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:06,349 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:06,728 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:07,730 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:08,732 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:08,821 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:09,371 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:52:09,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:52:09,698 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:52:09,741 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:10,350 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:10,731 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:11,736 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:12,734 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:13,739 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:14,707 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:14,736 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:15,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:16,739 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:17,746 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:18,746 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:19,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:20,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:20,741 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:21,354 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:52:21,606 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/8p4tfv7o-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:52:21,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:21,946 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:52:21,946 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:52:21,946 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:52:22,354 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:22,355 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:52:22,746 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:23,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:24,355 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:24,698 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:52:24,698 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:52:24,748 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:25,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:26,356 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:26,495 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:26,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:27,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:28,357 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:28,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:29,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:30,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:31,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:31,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:32,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:33,743 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:34,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:35,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:36,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:37,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:37,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:38,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:39,371 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:52:39,695 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:52:39,696 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:52:39,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:40,745 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:41,745 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:42,745 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:42,811 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:43,745 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:44,746 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:45,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:46,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:47,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:48,706 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:48,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:49,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:50,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:51,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:52,748 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:53,606 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:52:53,606 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_209651fa62317471e2ae.mp4 with policy now |
| 2025-01-07 21:52:53,609 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:52:53,610 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:52:53,610 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:52:53,610 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:52:53,751 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:53,870 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/wqvd6pty-media/videos/eval/3_eval_3_209651fa62317471e2ae.mp4 |
| 2025-01-07 21:52:54,365 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:54,365 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:52:54,365 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/3_eval_3_209651fa62317471e2ae.mp4 |
| 2025-01-07 21:52:54,366 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:52:54,639 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:54,662 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/c9y1qb0c-media/videos/eval/3_eval_3_209651fa62317471e2ae.mp4 |
| 2025-01-07 21:52:54,703 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:52:54,704 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:52:54,753 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:55,365 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:55,755 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:56,366 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:56,760 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:57,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:58,367 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:52:58,757 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:52:59,714 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:52:59,756 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:00,752 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:01,753 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:02,368 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:02,755 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:03,758 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:04,758 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:05,709 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:05,770 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:06,369 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:53:06,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:07,004 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:53:07,004 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:53:07,005 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:53:07,370 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:53:07,769 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:08,370 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:08,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:09,371 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:53:09,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:53:09,704 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:53:09,767 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:10,761 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:11,506 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:11,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:12,371 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:12,761 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:13,761 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:14,761 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:15,761 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:16,707 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:16,761 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:17,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:18,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:19,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:20,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:21,707 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:21,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:22,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:23,762 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:24,699 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:53:24,699 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:53:24,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:25,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:26,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:26,837 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:27,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:28,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:29,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:30,763 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:31,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:32,708 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:32,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:33,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:34,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:35,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:36,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:37,768 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:38,004 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:38,384 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:38,692 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:53:38,692 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_83ae6c438be885a5e86e.mp4 with policy now |
| 2025-01-07 21:53:38,697 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:53:38,697 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:53:38,698 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:53:38,698 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:53:38,771 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:38,949 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/k3ukfryr-media/videos/eval/4_eval_4_83ae6c438be885a5e86e.mp4 |
| 2025-01-07 21:53:39,372 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:53:39,385 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:39,385 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:53:39,385 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/4_eval_4_83ae6c438be885a5e86e.mp4 |
| 2025-01-07 21:53:39,385 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:53:39,675 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/7c94hrg0-media/videos/eval/4_eval_4_83ae6c438be885a5e86e.mp4 |
| 2025-01-07 21:53:39,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:53:39,702 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:53:39,777 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:40,385 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:40,774 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:41,790 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:42,782 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:43,077 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:43,796 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:44,386 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:44,788 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:45,795 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:46,387 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:46,784 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:47,783 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:48,790 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:48,870 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:49,788 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:50,056 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:53:50,057 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:53:50,057 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:53:50,389 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:53:50,389 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:50,389 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:53:50,648 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/8k5znsay-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:53:50,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:51,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:52,389 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:52,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:53,789 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:54,390 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:54,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:53:54,700 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:53:54,781 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:53:54,789 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:55,789 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:56,391 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:53:56,790 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:57,790 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:58,790 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:53:59,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:00,392 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:00,710 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:00,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:01,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:02,791 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:03,792 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:04,793 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:05,710 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:05,793 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:06,793 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:07,797 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:08,793 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:09,372 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:54:09,700 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:54:09,700 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:54:09,794 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:10,795 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:10,850 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:11,795 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:12,795 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:13,796 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:14,797 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:15,797 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:16,711 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:16,797 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:17,797 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:18,798 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:19,798 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:20,805 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:21,677 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:54:21,677 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_5c374bcb2343831d6c96.mp4 with policy now |
| 2025-01-07 21:54:21,683 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:54:21,684 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:54:21,684 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:54:21,684 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:54:21,726 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:21,807 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:21,936 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/2l8065zn-media/videos/eval/5_eval_5_5c374bcb2343831d6c96.mp4 |
| 2025-01-07 21:54:22,404 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:22,404 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:54:22,404 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/5_eval_5_5c374bcb2343831d6c96.mp4 |
| 2025-01-07 21:54:22,404 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:54:22,686 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/crb62mbz-media/videos/eval/5_eval_5_5c374bcb2343831d6c96.mp4 |
| 2025-01-07 21:54:22,805 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:23,805 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:24,405 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:24,711 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:54:24,711 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:54:24,803 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:25,812 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:26,405 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:26,806 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:27,082 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:27,807 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:28,808 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:29,809 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:30,407 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:30,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:31,815 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:32,716 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:32,816 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:33,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:34,818 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:35,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:36,112 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:54:36,112 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:54:36,113 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:54:36,409 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:54:36,409 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:54:36,828 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:37,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:38,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:38,410 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:38,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:39,373 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:54:39,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:54:39,702 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:54:39,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:40,825 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:41,869 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:42,411 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:42,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:43,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:44,174 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:44,412 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:54:44,820 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:45,821 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:46,821 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:47,821 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:48,821 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:49,717 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:49,821 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:50,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:51,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:52,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:53,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:54,701 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:54:54,702 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:54:54,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:54,827 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:54:55,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:56,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:57,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:58,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:54:59,822 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:00,717 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:00,823 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:01,823 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:02,823 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:03,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:04,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:05,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:06,717 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:06,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:07,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:08,421 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:08,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:09,373 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:55:09,722 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:55:09,722 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:55:09,732 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:55:09,824 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:55:09,824 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_1fd0a12988dce00dfb92.mp4 with policy now |
| 2025-01-07 21:55:09,824 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:55:09,825 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:55:09,825 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:09,825 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:55:10,083 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/r120ktgi-media/videos/eval/6_eval_6_1fd0a12988dce00dfb92.mp4 |
| 2025-01-07 21:55:10,421 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:10,422 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:55:10,422 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/6_eval_6_1fd0a12988dce00dfb92.mp4 |
| 2025-01-07 21:55:10,422 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:55:10,699 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/tbzwhest-media/videos/eval/6_eval_6_1fd0a12988dce00dfb92.mp4 |
| 2025-01-07 21:55:10,833 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:11,826 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:11,828 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:12,422 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:12,841 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:13,849 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:14,423 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:14,849 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:15,852 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:16,851 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:17,161 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:17,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:18,425 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:18,850 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:19,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:20,167 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:55:20,168 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:55:20,168 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:55:20,426 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:55:20,426 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:55:20,682 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/lhd51kd1-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:55:20,857 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:21,853 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:22,233 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:22,427 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:22,855 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:23,866 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:24,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:55:24,702 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:55:24,853 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:25,853 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:26,429 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:26,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:27,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:27,922 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:28,853 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:29,853 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:30,429 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:30,853 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:31,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:32,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:33,720 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:33,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:34,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:35,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:36,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:37,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:38,720 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:38,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:39,374 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:55:39,702 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:55:39,702 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:55:39,855 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:40,855 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:41,861 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:42,855 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:43,803 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:43,855 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:44,855 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:45,857 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:46,857 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:47,856 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:48,856 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:49,721 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:49,857 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:50,857 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:51,869 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:52,437 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:52,478 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:55:52,478 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_8ba49b1c68cf3a00e67a.mp4 with policy now |
| 2025-01-07 21:55:52,482 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:55:52,483 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:55:52,483 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:55:52,484 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:55:52,735 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/tz4dzcsu-media/videos/eval/7_eval_7_8ba49b1c68cf3a00e67a.mp4 |
| 2025-01-07 21:55:52,866 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:53,438 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:55:53,438 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/7_eval_7_8ba49b1c68cf3a00e67a.mp4 |
| 2025-01-07 21:55:53,438 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:55:53,739 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/jrod8liv-media/videos/eval/7_eval_7_8ba49b1c68cf3a00e67a.mp4 |
| 2025-01-07 21:55:53,869 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:54,438 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:54,709 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:55:54,709 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:55:54,798 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:55:54,866 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:55,874 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:56,888 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:57,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:58,440 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:55:58,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:55:59,878 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:00,440 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:00,738 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:00,878 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:01,879 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:02,877 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:03,881 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:04,224 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:56:04,224 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:56:04,225 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:56:04,442 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:56:04,442 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:04,443 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:56:04,881 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:05,900 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:06,442 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:06,726 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:06,891 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:07,882 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:08,443 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:08,882 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:09,374 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:56:09,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:56:09,704 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:56:09,882 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:10,444 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:10,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:11,842 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:11,882 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:12,882 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:13,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:14,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:15,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:16,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:17,725 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:17,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:18,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:19,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:20,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:21,883 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:22,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:23,723 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:23,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:24,704 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:56:24,704 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:56:24,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:25,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:26,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:27,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:28,796 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:28,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:29,884 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:30,885 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:31,885 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:32,885 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:33,885 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:34,532 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:34,901 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:35,771 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:56:35,772 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_d2cf0a27b6e7a9a0aea7.mp4 with policy now |
| 2025-01-07 21:56:35,786 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:56:35,787 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:56:35,787 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:56:35,787 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:56:35,902 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:36,028 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/8oizt1zu-media/videos/eval/8_eval_8_d2cf0a27b6e7a9a0aea7.mp4 |
| 2025-01-07 21:56:36,454 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:36,454 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:56:36,454 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/8_eval_8_d2cf0a27b6e7a9a0aea7.mp4 |
| 2025-01-07 21:56:36,454 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:56:36,777 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/p0lzpr9w-media/videos/eval/8_eval_8_d2cf0a27b6e7a9a0aea7.mp4 |
| 2025-01-07 21:56:36,906 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:37,918 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:38,454 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:38,918 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:39,374 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:56:39,730 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:56:39,730 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:56:39,795 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:39,913 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:40,455 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:40,918 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:41,919 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:42,918 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:43,919 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:44,456 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:44,919 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:45,739 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:45,919 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:46,918 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:47,927 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:48,279 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:56:48,279 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:56:48,280 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:56:48,458 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:56:48,458 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:56:48,714 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/2won4f61-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:56:48,939 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:49,942 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:50,459 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:50,740 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:50,943 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:51,941 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:52,955 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:53,944 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:54,460 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:54,734 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:56:54,734 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:56:54,951 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:55,810 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:56:55,961 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:56,944 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:57,944 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:58,462 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:56:58,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:56:59,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:00,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:01,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:01,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:02,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:03,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:04,945 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:05,946 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:06,946 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:07,740 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:07,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:08,946 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:09,375 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:57:09,718 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:57:09,718 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:57:09,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:10,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:11,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:12,786 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:12,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:13,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:14,468 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:14,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:15,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:16,469 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:16,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:17,947 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:18,521 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:18,949 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:19,950 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:20,949 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:21,949 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:22,949 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:23,739 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:23,950 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:24,472 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:24,740 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:57:24,740 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:57:24,953 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:25,132 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:57:25,132 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_eda3ab83a50fa0b2dcdc.mp4 with policy now |
| 2025-01-07 21:57:25,138 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:57:25,138 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:57:25,139 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:57:25,139 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:57:25,394 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/rs8mej0e-media/videos/eval/9_eval_9_eda3ab83a50fa0b2dcdc.mp4 |
| 2025-01-07 21:57:25,473 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:57:25,473 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/9_eval_9_eda3ab83a50fa0b2dcdc.mp4 |
| 2025-01-07 21:57:25,473 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:57:25,759 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/3gxbrza1-media/videos/eval/9_eval_9_eda3ab83a50fa0b2dcdc.mp4 |
| 2025-01-07 21:57:25,964 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:26,473 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:26,970 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:27,959 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:28,474 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:28,799 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:28,997 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:29,983 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:30,474 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:30,991 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:31,979 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:32,476 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:32,992 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:33,993 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:34,751 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:34,984 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:36,001 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:36,333 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:57:36,333 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:57:36,334 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:57:36,477 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:57:36,477 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:36,478 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:57:37,017 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:38,009 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:38,478 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:39,012 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:39,375 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:57:39,729 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:57:39,729 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:57:39,874 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:40,018 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:40,479 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:41,036 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:42,011 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:43,011 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:44,017 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:44,485 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:45,013 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:45,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:46,013 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:46,486 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:57:47,013 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:48,013 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:49,013 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:50,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:51,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:51,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:52,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:53,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:54,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:54,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:57:54,719 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:57:55,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:56,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:56,864 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:57:57,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:58,014 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:57:59,015 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:00,015 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:01,015 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:02,015 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:02,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:03,015 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:04,015 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:05,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:06,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:07,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:07,742 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:08,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:09,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:09,376 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:58:09,719 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:58:09,719 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:58:10,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:11,016 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:12,044 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:12,495 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:12,661 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:58:12,662 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_cc846afee20f9965cbd4.mp4 with policy now |
| 2025-01-07 21:58:12,662 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:58:12,662 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:58:12,662 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:58:12,663 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:58:12,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:12,913 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/yv4ldzu5-media/videos/eval/10_eval_10_cc846afee20f9965cbd4.mp4 |
| 2025-01-07 21:58:13,051 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:13,496 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:58:13,496 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/10_eval_10_cc846afee20f9965cbd4.mp4 |
| 2025-01-07 21:58:13,496 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:58:13,796 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/7o9ep2ct-media/videos/eval/10_eval_10_cc846afee20f9965cbd4.mp4 |
| 2025-01-07 21:58:14,051 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:14,496 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:15,072 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:16,065 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:17,073 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:18,071 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:18,173 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:18,498 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:19,072 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:20,081 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:20,499 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:21,109 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:22,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:23,076 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:23,710 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:24,102 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:24,394 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:58:24,394 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:58:24,394 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:58:24,500 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:58:24,501 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:24,501 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:58:24,730 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:58:24,730 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:58:24,763 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/sdfhlm89-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:58:25,104 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:26,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:26,501 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:27,087 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:28,083 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:28,502 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:28,949 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:29,097 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:30,101 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:30,502 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:31,094 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:32,105 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:32,504 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:33,082 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:34,082 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:34,247 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:34,505 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:35,082 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:36,083 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:36,506 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:37,083 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:38,083 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:38,506 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:39,083 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:39,376 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:58:39,377 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:39,724 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:58:39,724 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:58:40,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:40,507 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:41,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:42,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:42,508 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:43,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:44,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:44,488 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:45,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:46,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:47,084 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:48,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:48,512 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:49,086 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:49,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:50,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:50,513 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:58:51,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:52,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:53,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:54,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:54,724 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:58:54,724 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:58:54,834 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:58:55,085 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:56,086 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:57,086 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:58,086 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:59,086 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:58:59,877 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:00,103 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:00,518 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:00,643 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:59:00,643 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_bac8f9c871b47a881b70.mp4 with policy now |
| 2025-01-07 21:59:00,652 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:59:00,653 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:59:00,653 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:59:00,653 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:59:00,902 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/h2bw89nv-media/videos/eval/11_eval_11_bac8f9c871b47a881b70.mp4 |
| 2025-01-07 21:59:01,091 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:01,518 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:59:01,518 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/11_eval_11_bac8f9c871b47a881b70.mp4 |
| 2025-01-07 21:59:01,518 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:59:01,828 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/l7gc7soe-media/videos/eval/11_eval_11_bac8f9c871b47a881b70.mp4 |
| 2025-01-07 21:59:02,096 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:02,518 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:03,104 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:04,129 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:05,122 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:05,773 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:06,114 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:06,520 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:07,123 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:08,132 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:08,521 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:09,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:09,377 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:59:09,737 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:59:09,738 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:59:10,127 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:10,798 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:11,141 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:12,123 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:12,464 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:59:12,464 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:59:12,464 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:59:12,522 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 21:59:12,522 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:12,523 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:59:13,125 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:14,137 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:14,523 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:15,133 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:16,031 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:16,132 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:17,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:18,162 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:18,525 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:19,150 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:20,149 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:21,176 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:21,777 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:22,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:23,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:24,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:24,527 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:24,724 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:59:24,725 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:59:25,135 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:26,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:26,847 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:27,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:28,134 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:29,135 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:30,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:31,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:32,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:32,766 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:33,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:34,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:35,142 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:36,143 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:36,531 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:37,143 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:37,766 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:38,143 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:39,147 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:39,377 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 21:59:39,724 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:59:39,725 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:59:40,143 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:41,143 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:42,148 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:42,854 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:43,147 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:44,147 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:45,148 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:46,148 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:46,539 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:47,148 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:48,148 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:48,225 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:49,148 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:50,171 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:50,541 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:50,665 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 21:59:50,665 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_81d8d04ea7edc7b1c818.mp4 with policy now |
| 2025-01-07 21:59:50,669 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 21:59:50,669 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 21:59:50,670 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 21:59:50,670 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 21:59:50,930 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/67y65hit-media/videos/eval/12_eval_12_81d8d04ea7edc7b1c818.mp4 |
| 2025-01-07 21:59:51,165 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:51,541 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 21:59:51,542 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/12_eval_12_81d8d04ea7edc7b1c818.mp4 |
| 2025-01-07 21:59:51,542 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 21:59:51,874 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/4edgdnp9-media/videos/eval/12_eval_12_81d8d04ea7edc7b1c818.mp4 |
| 2025-01-07 21:59:52,179 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:52,542 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:53,198 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:53,764 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 21:59:54,197 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:54,746 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 21:59:54,746 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 21:59:55,215 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:56,199 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:56,543 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:57,225 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:58,224 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:58,544 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 21:59:59,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 21:59:59,581 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:00,235 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:01,213 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:02,217 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:02,520 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:00:02,520 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:00:02,521 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:00:02,546 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:00:02,546 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:02,546 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:00:03,215 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:04,201 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:04,547 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:04,814 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:05,244 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:06,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:07,214 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:08,218 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:08,548 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:09,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:09,378 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 22:00:09,739 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:00:09,739 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:00:09,894 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:10,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:11,233 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:12,232 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:13,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:14,220 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:14,551 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:15,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:15,799 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:16,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:17,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:18,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:19,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:20,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:20,801 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:21,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:22,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:23,221 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:24,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:24,732 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:00:24,732 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:00:25,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:25,863 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:26,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:27,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:28,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:29,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:30,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:31,222 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:31,800 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:32,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:33,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:34,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:35,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:36,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:36,800 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:37,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:38,224 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:39,223 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:39,378 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 22:00:39,765 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:00:39,765 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:00:40,245 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:40,561 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:41,022 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 22:00:41,023 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_eac1e3a9ea31f7ab5b3f.mp4 with policy now |
| 2025-01-07 22:00:41,023 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 22:00:41,023 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:00:41,023 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:00:41,024 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:00:41,231 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:41,279 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/9wqjt18f-media/videos/eval/13_eval_13_eac1e3a9ea31f7ab5b3f.mp4 |
| 2025-01-07 22:00:41,561 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:00:41,562 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/13_eval_13_eac1e3a9ea31f7ab5b3f.mp4 |
| 2025-01-07 22:00:41,562 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 22:00:41,850 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/ta0ravul-media/videos/eval/13_eval_13_eac1e3a9ea31f7ab5b3f.mp4 |
| 2025-01-07 22:00:42,131 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:42,238 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:42,562 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:43,245 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:44,238 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:45,264 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:46,264 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:46,563 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:47,264 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:47,628 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:48,265 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:48,564 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:49,290 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:50,286 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:51,280 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:52,294 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:52,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:53,283 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:54,281 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:54,749 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:00:54,749 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:00:55,304 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:56,291 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:57,290 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:58,283 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:58,568 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:00:58,569 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:00:58,595 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:00:58,595 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:00:58,595 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:00:58,595 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:00:58,823 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/3zwgkeg1-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:00:59,300 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:00:59,569 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:01:00,310 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:00,569 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:01,320 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:02,293 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:02,570 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:03,295 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:04,053 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:04,304 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:05,308 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:06,315 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:07,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:08,300 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:08,573 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:09,300 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:09,379 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 22:01:09,379 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:09,744 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:01:09,745 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:01:10,300 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:11,300 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:12,300 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:13,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:14,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:14,809 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:15,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:16,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:17,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:18,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:19,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:19,810 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:20,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:21,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:22,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:23,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:24,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:24,579 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:24,745 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:01:24,745 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:01:24,898 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:25,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:26,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:27,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:28,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:29,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:30,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:30,810 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:31,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:32,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:33,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:34,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:35,307 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:35,672 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 22:01:35,672 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_0dac4d6100436c1334e4.mp4 with policy now |
| 2025-01-07 22:01:35,682 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 22:01:35,682 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:01:35,682 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:35,682 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:35,939 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/d31nmdix-media/videos/eval/14_eval_14_0dac4d6100436c1334e4.mp4 |
| 2025-01-07 22:01:36,357 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:36,584 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:36,584 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:01:36,584 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/14_eval_14_0dac4d6100436c1334e4.mp4 |
| 2025-01-07 22:01:36,584 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval |
| 2025-01-07 22:01:36,776 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:36,893 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/ib39peo4-media/videos/eval/14_eval_14_0dac4d6100436c1334e4.mp4 |
| 2025-01-07 22:01:37,348 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:38,342 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:38,585 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:39,346 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:39,379 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 22:01:39,751 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:01:39,751 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:01:40,361 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:40,585 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:41,388 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:41,893 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:42,409 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:43,354 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:44,350 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:44,587 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:45,357 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:46,351 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:46,588 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:01:46,588 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:46,652 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:01:46,653 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:46,653 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:47,362 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:47,488 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 22:01:47,488 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 22:01:47,591 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:01:47,824 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 22:01:47,959 INFO wandb-upload_0:2237540 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpxqm9j9pt |
| 2025-01-07 22:01:48,268 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-07 22:01:48,277 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-07 22:01:48,352 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:48,399 INFO SenderThread:2237540 [sender.py:send_request_log_artifact():1525] logged artifact run-7u1dchof-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjUyMjE3Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTg3ODU0NA==', 'latestArtifact': None}} |
| 2025-01-07 22:01:48,400 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 22:01:48,400 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_15_24d8ed5706c93157f17a.table.json with policy now |
| 2025-01-07 22:01:48,400 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:01:48,400 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,401 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:48,401 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,401 DEBUG SenderThread:2237540 [sender.py:send():379] send: telemetry |
| 2025-01-07 22:01:48,401 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,402 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,402 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,402 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,402 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,404 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,404 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,405 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,405 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,405 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,405 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,406 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,406 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,406 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,406 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,407 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,407 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,407 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,408 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,408 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,408 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:48,409 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:48,409 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-07 22:01:48,589 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/sft7io99-media/table/eval/success_rate_ci_15_24d8ed5706c93157f17a.table.json |
| 2025-01-07 22:01:48,592 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:01:48,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:01:48,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/table/eval/success_rate_ci_15_24d8ed5706c93157f17a.table.json |
| 2025-01-07 22:01:48,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/table/eval/total_success_rate_ci_16_706adff9374d8bd6f9a2.table.json |
| 2025-01-07 22:01:48,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media |
| 2025-01-07 22:01:48,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/table |
| 2025-01-07 22:01:48,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/table/eval |
| 2025-01-07 22:01:48,836 INFO wandb-upload_2:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/1mfgcnas-media/table/eval/total_success_rate_ci_16_706adff9374d8bd6f9a2.table.json |
| 2025-01-07 22:01:48,843 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/gbfljdtf-media/table/eval/success_rate_ci_15_24d8ed5706c93157f17a.table.json |
| 2025-01-07 22:01:48,854 INFO wandb-upload_0:2237540 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpxm4ae1ed |
| 2025-01-07 22:01:49,273 INFO SenderThread:2237540 [sender.py:send_request_log_artifact():1525] logged artifact run-7u1dchof-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQyNjUyMjE5NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUzMTg3ODU0OA==', 'latestArtifact': None}} |
| 2025-01-07 22:01:49,273 DEBUG SenderThread:2237540 [sender.py:send():379] send: files |
| 2025-01-07 22:01:49,273 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_16_706adff9374d8bd6f9a2.table.json with policy now |
| 2025-01-07 22:01:49,273 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:01:49,273 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:01:49,274 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:01:49,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:49,533 INFO wandb-upload_2:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/gcn6k838-media/table/eval/total_success_rate_ci_16_706adff9374d8bd6f9a2.table.json |
| 2025-01-07 22:01:49,593 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:01:50,352 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:50,594 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:01:51,352 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:52,352 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:53,352 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:53,835 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:54,352 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:54,747 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-07 22:01:54,747 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-07 22:01:55,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:56,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:57,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:58,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:01:58,856 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:01:59,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:00,353 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:01,354 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:02,354 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:03,354 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:04,354 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:04,371 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:02:04,413 DEBUG SenderThread:2237540 [sender.py:send():379] send: exit |
| 2025-01-07 22:02:04,413 INFO SenderThread:2237540 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-07 22:02:04,414 INFO SenderThread:2237540 [sender.py:send_exit():588] handling runtime: 745 |
| 2025-01-07 22:02:04,414 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:02:04,414 INFO SenderThread:2237540 [sender.py:send_exit():594] send defer |
| 2025-01-07 22:02:04,414 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:04,414 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-07 22:02:04,414 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:04,414 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-07 22:02:04,415 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-07 22:02:04,415 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:04,415 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-07 22:02:04,415 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:04,415 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-07 22:02:04,415 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-07 22:02:04,415 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:04,415 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-07 22:02:04,415 INFO HandlerThread:2237540 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-07 22:02:04,415 INFO HandlerThread:2237540 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-07 22:02:04,415 DEBUG SystemMonitor:2237540 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-07 22:02:04,415 DEBUG SystemMonitor:2237540 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-07 22:02:04,416 INFO HandlerThread:2237540 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-07 22:02:04,469 INFO HandlerThread:2237540 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-07 22:02:04,469 INFO HandlerThread:2237540 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-07 22:02:04,470 INFO HandlerThread:2237540 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-07 22:02:04,470 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:04,470 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-07 22:02:04,470 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-07 22:02:04,470 DEBUG SenderThread:2237540 [sender.py:send():379] send: stats |
| 2025-01-07 22:02:04,470 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:04,470 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-07 22:02:04,471 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:04,471 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-07 22:02:04,471 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-07 22:02:04,471 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:04,471 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-07 22:02:04,600 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:02:06,601 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:02:09,602 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/config.yaml |
| 2025-01-07 22:02:10,661 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 22:02:10,661 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:02:10,661 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-07 22:02:10,662 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:10,662 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-07 22:02:10,662 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-07 22:02:10,662 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 22:02:10,663 DEBUG SenderThread:2237540 [sender.py:send():379] send: history |
| 2025-01-07 22:02:10,663 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-07 22:02:10,663 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:10,663 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-07 22:02:10,665 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:02:10,665 DEBUG SenderThread:2237540 [sender.py:send():379] send: summary |
| 2025-01-07 22:02:10,665 INFO SenderThread:2237540 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-07 22:02:10,665 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:10,665 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-07 22:02:10,665 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-07 22:02:10,665 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:10,666 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-07 22:02:10,666 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:10,666 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-07 22:02:10,666 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-07 22:02:10,666 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-07 22:02:10,666 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:10,666 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-07 22:02:10,666 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:10,666 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-07 22:02:11,415 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 22:02:11,603 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:02:13,835 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-07 22:02:13,835 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 22:02:13,836 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:13,836 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-07 22:02:13,836 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:13,836 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-07 22:02:13,836 INFO SenderThread:2237540 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-07 22:02:13,839 INFO SenderThread:2237540 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-07 22:02:13,840 INFO SenderThread:2237540 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-07 22:02:13,859 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-07 22:02:13,859 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:13,859 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-07 22:02:13,859 DEBUG SenderThread:2237540 [sender.py:send():379] send: artifact |
| 2025-01-07 22:02:14,416 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 22:02:14,541 INFO wandb-upload_1:2237540 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmph0mz9co_ |
| 2025-01-07 22:02:14,605 INFO Thread-12 :2237540 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:02:14,670 INFO wandb-upload_0:2237540 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpzwxnw11b |
| 2025-01-07 22:02:14,691 INFO wandb-upload_2:2237540 [upload_job.py:push():88] Uploaded file /tmp/tmpods0y954/wandb-job.json |
| 2025-01-07 22:02:15,296 INFO SenderThread:2237540 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQyNjUyMjY3OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQyNjE0MDMwNQ==', 'versionIndex': 437}}} |
| 2025-01-07 22:02:15,297 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:15,297 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-07 22:02:15,297 INFO SenderThread:2237540 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/config.yaml config.yaml |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json wandb-summary.json |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/requirements.txt requirements.txt |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log output.log |
| 2025-01-07 22:02:15,606 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/diff.patch diff.patch |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/2_eval_2_f1a3689635b40f332568.mp4 media/videos/eval/2_eval_2_f1a3689635b40f332568.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/6_eval_6_1fd0a12988dce00dfb92.mp4 media/videos/eval/6_eval_6_1fd0a12988dce00dfb92.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/12_eval_12_81d8d04ea7edc7b1c818.mp4 media/videos/eval/12_eval_12_81d8d04ea7edc7b1c818.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/10_eval_10_cc846afee20f9965cbd4.mp4 media/videos/eval/10_eval_10_cc846afee20f9965cbd4.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/3_eval_3_209651fa62317471e2ae.mp4 media/videos/eval/3_eval_3_209651fa62317471e2ae.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/9_eval_9_eda3ab83a50fa0b2dcdc.mp4 media/videos/eval/9_eval_9_eda3ab83a50fa0b2dcdc.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/4_eval_4_83ae6c438be885a5e86e.mp4 media/videos/eval/4_eval_4_83ae6c438be885a5e86e.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/11_eval_11_bac8f9c871b47a881b70.mp4 media/videos/eval/11_eval_11_bac8f9c871b47a881b70.mp4 |
| 2025-01-07 22:02:15,610 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/13_eval_13_eac1e3a9ea31f7ab5b3f.mp4 media/videos/eval/13_eval_13_eac1e3a9ea31f7ab5b3f.mp4 |
| 2025-01-07 22:02:15,611 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/0_eval_0_165af090a1eeabfb388b.mp4 media/videos/eval/0_eval_0_165af090a1eeabfb388b.mp4 |
| 2025-01-07 22:02:15,611 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/8_eval_8_d2cf0a27b6e7a9a0aea7.mp4 media/videos/eval/8_eval_8_d2cf0a27b6e7a9a0aea7.mp4 |
| 2025-01-07 22:02:15,611 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/7_eval_7_8ba49b1c68cf3a00e67a.mp4 media/videos/eval/7_eval_7_8ba49b1c68cf3a00e67a.mp4 |
| 2025-01-07 22:02:15,611 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/1_eval_1_a4095f09212ed4cd67c6.mp4 media/videos/eval/1_eval_1_a4095f09212ed4cd67c6.mp4 |
| 2025-01-07 22:02:15,611 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/14_eval_14_0dac4d6100436c1334e4.mp4 media/videos/eval/14_eval_14_0dac4d6100436c1334e4.mp4 |
| 2025-01-07 22:02:15,611 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/videos/eval/5_eval_5_5c374bcb2343831d6c96.mp4 media/videos/eval/5_eval_5_5c374bcb2343831d6c96.mp4 |
| 2025-01-07 22:02:15,617 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/table/eval/success_rate_ci_15_24d8ed5706c93157f17a.table.json media/table/eval/success_rate_ci_15_24d8ed5706c93157f17a.table.json |
| 2025-01-07 22:02:15,617 INFO SenderThread:2237540 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/media/table/eval/total_success_rate_ci_16_706adff9374d8bd6f9a2.table.json media/table/eval/total_success_rate_ci_16_706adff9374d8bd6f9a2.table.json |
| 2025-01-07 22:02:15,617 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-07 22:02:15,617 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 22:02:15,617 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:15,618 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-07 22:02:15,618 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:15,618 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-07 22:02:15,618 INFO SenderThread:2237540 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 22:02:15,790 INFO wandb-upload_0:2237540 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/wandb-summary.json |
| 2025-01-07 22:02:15,796 INFO wandb-upload_1:2237540 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/config.yaml |
| 2025-01-07 22:02:15,839 INFO wandb-upload_3:2237540 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/requirements.txt |
| 2025-01-07 22:02:15,877 INFO wandb-upload_2:2237540 [upload_job.py:push():130] Uploaded file /tmp/tmp07wha5auwandb/egceoa3o-events.out.tfevents.1736304579.leonmkim-ROG-Strix-G15CS-G15CS.2237375.0 |
| 2025-01-07 22:02:15,887 INFO wandb-upload_4:2237540 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/files/output.log |
| 2025-01-07 22:02:16,087 INFO Thread-11 (_thread_body):2237540 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-07 22:02:16,087 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:16,087 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-07 22:02:16,087 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:16,087 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-07 22:02:16,088 INFO SenderThread:2237540 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-07 22:02:16,090 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-07 22:02:16,090 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:16,090 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-07 22:02:16,091 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:16,091 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-07 22:02:16,091 INFO SenderThread:2237540 [file_stream.py:finish():601] file stream finish called |
| 2025-01-07 22:02:16,301 INFO SenderThread:2237540 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-07 22:02:16,301 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-07 22:02:16,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:16,301 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-07 22:02:16,301 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:16,301 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-07 22:02:16,301 INFO SenderThread:2237540 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-07 22:02:16,301 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-07 22:02:16,301 INFO HandlerThread:2237540 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-07 22:02:16,302 DEBUG SenderThread:2237540 [sender.py:send():379] send: final |
| 2025-01-07 22:02:16,302 DEBUG SenderThread:2237540 [sender.py:send():379] send: footer |
| 2025-01-07 22:02:16,302 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: defer |
| 2025-01-07 22:02:16,302 INFO SenderThread:2237540 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-07 22:02:16,302 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 22:02:16,302 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 22:02:16,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-07 22:02:16,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-07 22:02:16,303 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: server_info |
| 2025-01-07 22:02:16,303 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-07 22:02:16,305 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-07 22:02:16,305 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-07 22:02:16,363 DEBUG SenderThread:2237540 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-07 22:02:16,363 INFO MainThread:2237540 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-07 22:02:16,364 INFO MainThread:2237540 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-07 22:02:16,364 INFO MainThread:2237540 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-07 22:02:16,364 DEBUG HandlerThread:2237540 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-07 22:02:16,364 INFO HandlerThread:2237540 [handler.py:finish():882] shutting down handler |
| 2025-01-07 22:02:17,303 INFO WriterThread:2237540 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1002_0/7u1dchof/wandb/run-20250107_214939-7u1dchof/run-7u1dchof.wandb |
| 2025-01-07 22:02:17,363 INFO SenderThread:2237540 [sender.py:finish():1615] shutting down sender |
| 2025-01-07 22:02:17,363 INFO SenderThread:2237540 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-07 22:02:17,363 INFO SenderThread:2237540 [file_pusher.py:join():175] waiting for file pusher |
|
|
