| 2025-01-01 11:03:03,231 INFO StreamThr :1789646 [internal.py:wandb_internal():85] W&B internal server running at pid: 1789646, started at: 2025-01-01 11:03:03.230653 |
| 2025-01-01 11:03:03,232 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status |
| 2025-01-01 11:03:03,245 INFO WriterThread:1789646 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/run-hex7gk4r.wandb |
| 2025-01-01 11:03:03,246 DEBUG SenderThread:1789646 [sender.py:send():379] send: header |
| 2025-01-01 11:03:03,256 DEBUG SenderThread:1789646 [sender.py:send():379] send: run |
| 2025-01-01 11:03:03,488 INFO SenderThread:1789646 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files |
| 2025-01-01 11:03:03,488 INFO SenderThread:1789646 [sender.py:_start_run_threads():1188] run started: hex7gk4r with start time 1735747383.230972 |
| 2025-01-01 11:03:03,501 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-01 11:03:03,501 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: check_version |
| 2025-01-01 11:03:03,570 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-01 11:03:03,575 DEBUG HandlerThread:1789646 [system_info.py:__init__():26] System info init |
| 2025-01-01 11:03:03,575 DEBUG HandlerThread:1789646 [system_info.py:__init__():41] System info init done |
| 2025-01-01 11:03:03,575 INFO HandlerThread:1789646 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-01 11:03:03,576 INFO SystemMonitor:1789646 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-01 11:03:03,576 INFO HandlerThread:1789646 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-01 11:03:03,577 INFO SystemMonitor:1789646 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-01 11:03:03,577 INFO SystemMonitor:1789646 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-01 11:03:03,578 INFO SystemMonitor:1789646 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-01 11:03:03,578 INFO SystemMonitor:1789646 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-01 11:03:03,579 INFO SystemMonitor:1789646 [interfaces.py:start():188] Started network monitoring |
| 2025-01-01 11:03:03,590 DEBUG HandlerThread:1789646 [system_info.py:probe():152] Probing system |
| 2025-01-01 11:03:03,591 DEBUG HandlerThread:1789646 [system_info.py:_probe_git():137] Probing git |
| 2025-01-01 11:03:03,596 DEBUG HandlerThread:1789646 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-01 11:03:03,596 DEBUG HandlerThread:1789646 [system_info.py:probe():200] Probing system done |
| 2025-01-01 11:03:03,596 DEBUG HandlerThread:1789646 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T16:03:03.590194', 'startedAt': '2025-01-01T16:03:03.217444', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3561.171875, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.942, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 664.0837097167969}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-01 11:03:03,596 INFO HandlerThread:1789646 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-01 11:03:03,596 INFO HandlerThread:1789646 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-01 11:03:03,596 DEBUG HandlerThread:1789646 [system_info.py:_save_code():44] Saving code |
| 2025-01-01 11:03:03,598 DEBUG HandlerThread:1789646 [system_info.py:_save_code():67] Saving code done |
| 2025-01-01 11:03:03,598 DEBUG HandlerThread:1789646 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-01 11:03:03,659 DEBUG HandlerThread:1789646 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-01 11:03:03,659 INFO HandlerThread:1789646 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-01 11:03:03,662 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:03:03,662 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-01 11:03:03,663 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-01 11:03:03,663 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-01 11:03:03,901 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-01 11:03:03,901 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-01 11:03:03,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:03:03,902 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:03:03,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:03,903 INFO HandlerThread:1789646 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/tb" |
| save: true |
| } |
| control { |
| relay_id: "9383c060aecf4fb1bae46e64e533ff79" |
| } |
| _info { |
| stream_id: "hex7gk4r" |
| } |
|
|
| 2025-01-01 11:03:03,904 DEBUG HandlerThread:1789646 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-01 11:03:03,911 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/m540yw5f-wandb-metadata.json |
| 2025-01-01 11:03:03,917 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/5hwe7lky-code/FISH/eval_robot.py |
| 2025-01-01 11:03:03,917 INFO wandb-upload_2:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/r9xivxet-diff.patch |
| 2025-01-01 11:03:04,005 DEBUG SenderThread:1789646 [sender.py:send():379] send: telemetry |
| 2025-01-01 11:03:04,005 DEBUG SenderThread:1789646 [sender.py:send():379] send: tbrecord |
| 2025-01-01 11:03:04,005 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:03:04,005 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 with policy live |
| 2025-01-01 11:03:04,206 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/el9lws25-events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:03:04,489 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/requirements.txt |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/diff.patch |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/code/FISH/eval_robot.py |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-metadata.json |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/code/FISH |
| 2025-01-01 11:03:04,490 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/code |
| 2025-01-01 11:03:04,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:05,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:06,489 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:06,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:07,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:08,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:08,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:09,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:10,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:11,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:12,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:13,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:13,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:14,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:15,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:16,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:17,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:18,901 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:03:18,901 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:03:18,945 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:18,960 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:19,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:20,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:21,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:22,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:23,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:24,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:24,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:25,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:26,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:27,698 DEBUG SenderThread:1789646 [sender.py:send():379] send: config |
| 2025-01-01 11:03:27,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:28,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:29,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:29,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:30,495 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:30,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:31,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:32,496 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:32,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:33,901 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:03:33,902 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:03:33,945 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:34,496 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:34,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:34,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:35,496 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/config.yaml |
| 2025-01-01 11:03:35,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:36,497 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:36,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:37,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:38,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:39,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:40,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:40,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:41,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:42,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:43,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:44,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:45,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:45,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:46,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:47,907 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:48,901 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:03:48,902 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:03:48,945 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:49,907 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:50,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:51,653 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:51,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:52,501 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:52,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:53,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:54,502 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:54,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:55,909 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:56,502 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:03:56,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:03:56,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:57,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:58,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:03:59,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:00,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:01,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:01,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:02,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:03,579 DEBUG SystemMonitor:1789646 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-01 11:04:03,579 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:04:03,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:04:03,902 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:04:03,945 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:04,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:05,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:06,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:07,003 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:07,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:08,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:09,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:10,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:11,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:12,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:12,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:13,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:14,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:15,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:16,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:17,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:18,790 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:18,902 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:04:18,902 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:04:18,945 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:19,912 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:20,508 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:20,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:21,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:22,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:23,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:23,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:24,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:25,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:26,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:27,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:28,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:28,914 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:29,382 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:04:29,382 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:04:29,382 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_0a25a53b5c8820b669aa.mp4 with policy now |
| 2025-01-01 11:04:29,383 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:04:29,383 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:04:29,384 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:04:29,511 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/0_eval_0_0a25a53b5c8820b669aa.mp4 |
| 2025-01-01 11:04:29,511 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:04:29,511 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media |
| 2025-01-01 11:04:29,511 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval |
| 2025-01-01 11:04:29,511 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos |
| 2025-01-01 11:04:29,653 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/ob5kp906-media/videos/eval/0_eval_0_0a25a53b5c8820b669aa.mp4 |
| 2025-01-01 11:04:29,914 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:29,951 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/72hg7fzg-media/videos/eval/0_eval_0_0a25a53b5c8820b669aa.mp4 |
| 2025-01-01 11:04:30,511 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:30,914 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:31,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:32,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:33,580 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:04:33,904 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:04:33,904 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:04:33,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:34,005 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:34,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:35,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:36,513 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:36,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:37,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:38,514 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:38,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:39,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:39,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:40,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:41,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:42,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:43,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:44,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:45,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:45,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:46,916 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:47,917 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:48,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:04:48,903 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:04:48,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:49,917 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:50,913 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:50,961 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:51,518 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:04:51,759 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/l69935fl-events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:04:51,917 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:52,518 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:52,917 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:53,917 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:54,519 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:54,918 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:55,918 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:56,520 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:04:56,886 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:04:56,918 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:57,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:58,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:04:59,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:00,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:01,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:01,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:02,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:03,580 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:05:03,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:05:03,903 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:05:03,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:04,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:05,919 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:06,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:06,987 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:07,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:08,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:09,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:10,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:11,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:12,907 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:12,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:13,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:14,920 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:15,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:16,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:17,907 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:17,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:18,903 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:05:18,903 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:05:18,945 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:19,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:20,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:21,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:22,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:23,030 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:23,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:24,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:25,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:26,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:27,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:28,907 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:28,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:29,923 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:30,531 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:30,819 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:05:30,820 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_fae1d5a4084c3fa7e4ce.mp4 with policy now |
| 2025-01-01 11:05:30,820 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:05:30,821 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:05:30,821 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:05:30,821 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:05:30,923 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:31,081 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/aq2sgdsa-media/videos/eval/1_eval_1_fae1d5a4084c3fa7e4ce.mp4 |
| 2025-01-01 11:05:31,531 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:05:31,532 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/1_eval_1_fae1d5a4084c3fa7e4ce.mp4 |
| 2025-01-01 11:05:31,532 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval |
| 2025-01-01 11:05:31,807 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/3wtb49tu-media/videos/eval/1_eval_1_fae1d5a4084c3fa7e4ce.mp4 |
| 2025-01-01 11:05:31,923 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:32,532 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:32,925 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:33,532 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:33,581 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:05:33,909 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:05:33,910 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:05:33,953 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:34,067 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:34,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:35,925 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:36,533 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:36,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:37,925 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:38,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:39,908 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:39,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:40,534 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:40,535 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:05:40,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:41,474 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:05:41,474 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:05:41,474 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:05:41,535 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:05:41,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:42,535 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:42,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:43,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:44,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:45,395 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:45,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:46,536 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:46,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:47,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:48,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:05:48,905 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:05:48,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:49,926 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:50,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:50,965 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:51,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:52,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:53,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:54,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:55,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:56,578 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:05:56,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:57,927 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:58,540 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:05:58,931 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:05:59,928 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:00,541 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:00,928 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:01,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:01,928 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:02,928 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:03,581 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:06:03,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:06:03,905 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:06:03,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:04,929 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:05,929 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:06,929 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:07,022 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:07,929 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:08,929 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:09,929 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:10,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:11,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:12,910 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:12,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:13,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:14,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:15,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:16,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:17,911 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:17,930 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:18,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:06:18,905 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:06:18,953 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:19,933 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:20,454 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:06:20,454 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_07f9274f756bd81cbaee.mp4 with policy now |
| 2025-01-01 11:06:20,455 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:06:20,455 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:06:20,456 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:06:20,471 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:06:20,548 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:20,548 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:06:20,548 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/2_eval_2_07f9274f756bd81cbaee.mp4 |
| 2025-01-01 11:06:20,548 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval |
| 2025-01-01 11:06:20,708 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/okcuosdj-media/videos/eval/2_eval_2_07f9274f756bd81cbaee.mp4 |
| 2025-01-01 11:06:20,931 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:20,965 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/lfz8ukdf-media/videos/eval/2_eval_2_07f9274f756bd81cbaee.mp4 |
| 2025-01-01 11:06:21,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:22,548 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:22,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:23,476 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:23,549 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:23,936 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:24,942 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:25,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:26,550 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:26,940 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:27,937 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:28,918 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:28,938 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:29,942 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:30,235 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:06:30,235 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:06:30,235 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:06:30,551 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:30,551 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:06:30,551 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:06:30,785 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/0kxa3lse-events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:06:30,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:31,940 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:32,552 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:32,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:33,582 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:06:33,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:06:33,905 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:06:33,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:33,995 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:34,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:35,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:36,553 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:36,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:37,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:38,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:39,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:39,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:40,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:41,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:42,941 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:43,942 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:44,915 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:44,942 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:45,942 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:46,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:47,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:48,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:06:48,905 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:06:48,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:49,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:50,013 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:50,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:51,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:52,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:53,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:54,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:55,550 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:06:55,943 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:56,560 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:06:56,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:57,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:58,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:06:59,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:00,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:01,093 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:01,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:02,562 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:02,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:03,582 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:07:03,905 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:07:03,906 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:07:03,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:04,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:05,944 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:06,946 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:07,052 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:07,946 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:08,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:09,684 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:07:09,684 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_26ea14ca5e0897c83196.mp4 with policy now |
| 2025-01-01 11:07:09,687 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:07:09,688 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:07:09,688 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:07:09,734 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:07:09,933 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/m2mc05m1-media/videos/eval/3_eval_3_26ea14ca5e0897c83196.mp4 |
| 2025-01-01 11:07:09,949 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:10,565 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:10,565 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:07:10,565 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/3_eval_3_26ea14ca5e0897c83196.mp4 |
| 2025-01-01 11:07:10,565 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval |
| 2025-01-01 11:07:10,876 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/uxyl7jzq-media/videos/eval/3_eval_3_26ea14ca5e0897c83196.mp4 |
| 2025-01-01 11:07:10,950 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:11,953 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:12,735 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:12,953 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:13,960 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:14,962 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:15,957 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:16,567 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:16,960 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:17,964 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:18,009 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:18,567 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:18,917 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:07:18,917 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:07:18,962 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:19,966 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:20,960 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:21,960 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:22,961 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:23,841 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:23,956 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:24,302 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:07:24,302 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:07:24,302 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:07:24,569 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:07:24,569 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:07:24,964 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:25,969 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:26,570 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:26,971 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:27,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:28,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:28,963 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:29,975 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:30,572 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:30,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:31,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:32,572 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:32,966 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:33,583 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:07:33,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:07:33,906 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:07:33,966 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:34,015 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:34,574 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:34,966 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:35,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:36,574 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:36,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:37,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:38,575 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:07:38,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:39,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:39,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:40,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:41,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:42,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:43,967 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:44,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:45,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:45,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:46,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:47,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:48,906 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:07:48,906 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:07:48,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:49,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:50,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:50,969 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:51,968 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:52,969 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:53,971 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:54,971 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:55,971 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:56,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:07:56,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:57,971 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:58,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:07:59,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:00,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:01,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:02,921 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:02,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:03,583 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:08:03,907 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:08:03,907 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:08:03,972 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:04,973 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:05,973 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:06,973 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:07,977 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:08,407 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:08,587 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:08,839 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:08:08,839 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_20669f7dea9bec0e6b76.mp4 with policy now |
| 2025-01-01 11:08:08,841 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:08:08,842 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:08:08,842 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:08,842 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:08,978 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:09,094 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/5gyuhmzl-media/videos/eval/4_eval_4_20669f7dea9bec0e6b76.mp4 |
| 2025-01-01 11:08:09,587 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:08:09,587 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/4_eval_4_20669f7dea9bec0e6b76.mp4 |
| 2025-01-01 11:08:09,588 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval |
| 2025-01-01 11:08:09,869 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/2fj0xs0t-media/videos/eval/4_eval_4_20669f7dea9bec0e6b76.mp4 |
| 2025-01-01 11:08:09,981 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:10,587 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:10,988 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:11,995 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:12,984 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:13,514 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:13,996 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:14,589 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:14,988 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:15,994 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:16,993 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:17,993 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:18,590 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:18,922 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-01 11:08:18,923 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-01 11:08:18,992 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:18,992 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:19,996 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:20,995 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:21,995 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:23,014 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:24,005 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:24,159 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-01 11:08:24,159 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-01 11:08:24,298 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:08:24,592 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:24,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:08:24,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:08:24,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media |
| 2025-01-01 11:08:24,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table/eval |
| 2025-01-01 11:08:24,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table |
| 2025-01-01 11:08:24,650 INFO wandb-upload_1:1789646 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpnor_r9wg |
| 2025-01-01 11:08:24,693 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-01 11:08:24,701 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-01 11:08:24,769 INFO wandb-upload_2:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/8rhbygf3-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:08:24,841 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/izvtcfdg-events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:08:24,998 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:25,052 INFO SenderThread:1789646 [sender.py:send_request_log_artifact():1525] logged artifact run-hex7gk4r-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDY3NDAzMg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTI0MDIwNg==', 'latestArtifact': None}} |
| 2025-01-01 11:08:25,052 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:08:25,052 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json with policy now |
| 2025-01-01 11:08:25,053 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:25,053 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:08:25,053 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,054 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,055 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:08:25,055 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,058 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,059 DEBUG SenderThread:1789646 [sender.py:send():379] send: telemetry |
| 2025-01-01 11:08:25,059 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,059 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,059 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,059 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,060 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,060 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,060 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,060 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,060 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,061 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,061 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,061 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,061 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,062 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,062 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,062 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,062 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,063 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,063 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,063 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,063 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,063 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,064 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-01 11:08:25,297 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/8eoze2vw-media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:08:25,437 INFO wandb-upload_0:1789646 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpz1gbqcms |
| 2025-01-01 11:08:25,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:08:25,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:08:25,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table/eval/total_success_rate_ci_6_51a31336f9897b8b206f.table.json |
| 2025-01-01 11:08:25,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table/eval |
| 2025-01-01 11:08:25,804 INFO SenderThread:1789646 [sender.py:send_request_log_artifact():1525] logged artifact run-hex7gk4r-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDY3NDA0Mw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTI0MDIxMQ==', 'latestArtifact': None}} |
| 2025-01-01 11:08:25,804 DEBUG SenderThread:1789646 [sender.py:send():379] send: files |
| 2025-01-01 11:08:25,804 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_51a31336f9897b8b206f.table.json with policy now |
| 2025-01-01 11:08:25,804 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:08:25,804 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:25,805 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,805 DEBUG SenderThread:1789646 [sender.py:send():379] send: exit |
| 2025-01-01 11:08:25,805 INFO SenderThread:1789646 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-01 11:08:25,805 INFO SenderThread:1789646 [sender.py:send_exit():588] handling runtime: 321 |
| 2025-01-01 11:08:25,806 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:25,806 INFO SenderThread:1789646 [sender.py:send_exit():594] send defer |
| 2025-01-01 11:08:25,806 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:25,806 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-01 11:08:25,806 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:25,806 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-01 11:08:25,806 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-01 11:08:25,806 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:25,807 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-01 11:08:25,807 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:25,807 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-01 11:08:25,807 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-01 11:08:25,807 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:25,807 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-01 11:08:25,807 INFO HandlerThread:1789646 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-01 11:08:25,807 DEBUG SystemMonitor:1789646 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-01 11:08:25,807 INFO HandlerThread:1789646 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-01 11:08:25,807 DEBUG SystemMonitor:1789646 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-01 11:08:25,808 INFO HandlerThread:1789646 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-01 11:08:25,860 INFO HandlerThread:1789646 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-01 11:08:25,860 INFO HandlerThread:1789646 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-01 11:08:25,861 INFO HandlerThread:1789646 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-01 11:08:25,861 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:25,861 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-01 11:08:25,861 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-01 11:08:25,861 DEBUG SenderThread:1789646 [sender.py:send():379] send: stats |
| 2025-01-01 11:08:25,861 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:25,861 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-01 11:08:25,862 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:25,862 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-01 11:08:25,862 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-01 11:08:25,862 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:25,862 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-01 11:08:25,976 INFO wandb-upload_2:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/fwtuz5zp-media/table/eval/total_success_rate_ci_6_51a31336f9897b8b206f.table.json |
| 2025-01-01 11:08:26,223 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/6n7joruu-media/table/eval/total_success_rate_ci_6_51a31336f9897b8b206f.table.json |
| 2025-01-01 11:08:26,593 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:26,594 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:08:31,595 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/config.yaml |
| 2025-01-01 11:08:32,380 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:08:32,380 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:32,380 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-01 11:08:32,380 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:32,380 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-01 11:08:32,380 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-01 11:08:32,380 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:08:32,381 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:32,382 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-01 11:08:32,382 DEBUG SenderThread:1789646 [sender.py:send():379] send: history |
| 2025-01-01 11:08:32,382 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-01 11:08:32,383 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:32,383 DEBUG SenderThread:1789646 [sender.py:send():379] send: summary |
| 2025-01-01 11:08:32,387 INFO SenderThread:1789646 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-01 11:08:32,387 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:32,387 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-01 11:08:32,387 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-01 11:08:32,387 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:32,387 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-01 11:08:32,387 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:32,387 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-01 11:08:32,388 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-01 11:08:32,388 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-01 11:08:32,388 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:32,388 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-01 11:08:32,388 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:32,388 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-01 11:08:32,596 INFO Thread-12 :1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:08:33,227 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:08:34,935 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-01 11:08:34,935 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:08:34,936 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:34,936 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-01 11:08:34,936 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:34,936 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-01 11:08:34,936 INFO SenderThread:1789646 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-01 11:08:34,938 INFO SenderThread:1789646 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-01 11:08:34,940 INFO SenderThread:1789646 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-01 11:08:34,949 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-01 11:08:34,950 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:34,950 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-01 11:08:34,950 DEBUG SenderThread:1789646 [sender.py:send():379] send: artifact |
| 2025-01-01 11:08:35,133 INFO SenderThread:1789646 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDIyMjk2Ng==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDY2NzA0NA==', 'versionIndex': 404}}} |
| 2025-01-01 11:08:35,133 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:35,133 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-01 11:08:35,133 INFO SenderThread:1789646 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-01 11:08:35,227 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:08:35,597 INFO SenderThread:1789646 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:35,597 INFO SenderThread:1789646 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files |
| 2025-01-01 11:08:35,597 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/config.yaml config.yaml |
| 2025-01-01 11:08:35,597 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json wandb-summary.json |
| 2025-01-01 11:08:35,597 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:08:35,598 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/requirements.txt requirements.txt |
| 2025-01-01 11:08:35,598 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-01 11:08:35,598 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log output.log |
| 2025-01-01 11:08:35,600 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/diff.patch diff.patch |
| 2025-01-01 11:08:35,602 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-01 11:08:35,606 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/3_eval_3_26ea14ca5e0897c83196.mp4 media/videos/eval/3_eval_3_26ea14ca5e0897c83196.mp4 |
| 2025-01-01 11:08:35,606 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/4_eval_4_20669f7dea9bec0e6b76.mp4 media/videos/eval/4_eval_4_20669f7dea9bec0e6b76.mp4 |
| 2025-01-01 11:08:35,606 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/1_eval_1_fae1d5a4084c3fa7e4ce.mp4 media/videos/eval/1_eval_1_fae1d5a4084c3fa7e4ce.mp4 |
| 2025-01-01 11:08:35,606 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/0_eval_0_0a25a53b5c8820b669aa.mp4 media/videos/eval/0_eval_0_0a25a53b5c8820b669aa.mp4 |
| 2025-01-01 11:08:35,606 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/videos/eval/2_eval_2_07f9274f756bd81cbaee.mp4 media/videos/eval/2_eval_2_07f9274f756bd81cbaee.mp4 |
| 2025-01-01 11:08:35,606 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table/eval/total_success_rate_ci_6_51a31336f9897b8b206f.table.json media/table/eval/total_success_rate_ci_6_51a31336f9897b8b206f.table.json |
| 2025-01-01 11:08:35,607 INFO SenderThread:1789646 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json media/table/eval/success_rate_ci_5_1b420c32327aa0ba7f75.table.json |
| 2025-01-01 11:08:35,607 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-01 11:08:35,607 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:08:35,607 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:35,607 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-01 11:08:35,607 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:35,607 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-01 11:08:35,607 INFO SenderThread:1789646 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-01 11:08:35,776 INFO wandb-upload_1:1789646 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/wandb-summary.json |
| 2025-01-01 11:08:35,781 INFO wandb-upload_2:1789646 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/config.yaml |
| 2025-01-01 11:08:35,828 INFO wandb-upload_3:1789646 [upload_job.py:push():130] Uploaded file /tmp/tmpkhe9yb83wandb/lj4wjqap-events.out.tfevents.1735747383.leonmkim-ROG-Strix-G15CS-G15CS.1789455.0 |
| 2025-01-01 11:08:35,833 INFO wandb-upload_4:1789646 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/output.log |
| 2025-01-01 11:08:35,834 INFO wandb-upload_0:1789646 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/files/requirements.txt |
| 2025-01-01 11:08:36,034 INFO Thread-11 (_thread_body):1789646 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-01 11:08:36,034 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:36,035 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-01 11:08:36,035 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:36,035 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-01 11:08:36,035 INFO SenderThread:1789646 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-01 11:08:36,036 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-01 11:08:36,036 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:36,036 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-01 11:08:36,036 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:36,037 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-01 11:08:36,037 INFO SenderThread:1789646 [file_stream.py:finish():601] file stream finish called |
| 2025-01-01 11:08:36,228 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:08:36,243 INFO SenderThread:1789646 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-01 11:08:36,243 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-01 11:08:36,243 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:08:36,243 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:36,243 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-01 11:08:36,244 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:36,244 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-01 11:08:36,244 INFO SenderThread:1789646 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-01 11:08:36,244 DEBUG SenderThread:1789646 [sender.py:send():379] send: final |
| 2025-01-01 11:08:36,244 DEBUG SenderThread:1789646 [sender.py:send():379] send: footer |
| 2025-01-01 11:08:36,244 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-01 11:08:36,244 INFO HandlerThread:1789646 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-01 11:08:36,244 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: defer |
| 2025-01-01 11:08:36,244 INFO SenderThread:1789646 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-01 11:08:36,245 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:08:36,245 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:08:36,245 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-01 11:08:36,245 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-01 11:08:36,246 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-01 11:08:36,246 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: server_info |
| 2025-01-01 11:08:36,247 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-01 11:08:36,248 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-01 11:08:36,296 DEBUG SenderThread:1789646 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-01 11:08:36,296 INFO MainThread:1789646 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-01 11:08:36,296 INFO MainThread:1789646 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-01 11:08:36,297 INFO MainThread:1789646 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-01 11:08:36,297 DEBUG HandlerThread:1789646 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-01 11:08:36,297 INFO HandlerThread:1789646 [handler.py:finish():882] shutting down handler |
| 2025-01-01 11:08:37,246 INFO WriterThread:1789646 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1006_0/hex7gk4r/wandb/run-20250101_110303-hex7gk4r/run-hex7gk4r.wandb |
| 2025-01-01 11:08:37,296 INFO SenderThread:1789646 [sender.py:finish():1615] shutting down sender |
| 2025-01-01 11:08:37,296 INFO SenderThread:1789646 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-01 11:08:37,296 INFO SenderThread:1789646 [file_pusher.py:join():175] waiting for file pusher |
|
|
