| 2025-01-03 20:09:36,994 INFO StreamThr :1963622 [internal.py:wandb_internal():85] W&B internal server running at pid: 1963622, started at: 2025-01-03 20:09:36.993796 |
| 2025-01-03 20:09:36,995 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status |
| 2025-01-03 20:09:37,008 INFO WriterThread:1963622 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/run-tmmu86ea.wandb |
| 2025-01-03 20:09:37,008 DEBUG SenderThread:1963622 [sender.py:send():379] send: header |
| 2025-01-03 20:09:37,016 DEBUG SenderThread:1963622 [sender.py:send():379] send: run |
| 2025-01-03 20:09:37,250 INFO SenderThread:1963622 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files |
| 2025-01-03 20:09:37,250 INFO SenderThread:1963622 [sender.py:_start_run_threads():1188] run started: tmmu86ea with start time 1735952976.995053 |
| 2025-01-03 20:09:37,268 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-03 20:09:37,268 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: check_version |
| 2025-01-03 20:09:37,312 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-03 20:09:37,316 DEBUG HandlerThread:1963622 [system_info.py:__init__():26] System info init |
| 2025-01-03 20:09:37,316 DEBUG HandlerThread:1963622 [system_info.py:__init__():41] System info init done |
| 2025-01-03 20:09:37,316 INFO HandlerThread:1963622 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-03 20:09:37,317 INFO SystemMonitor:1963622 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-03 20:09:37,317 INFO HandlerThread:1963622 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-03 20:09:37,317 INFO SystemMonitor:1963622 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-03 20:09:37,318 INFO SystemMonitor:1963622 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-03 20:09:37,318 INFO SystemMonitor:1963622 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-03 20:09:37,319 INFO SystemMonitor:1963622 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-03 20:09:37,319 INFO SystemMonitor:1963622 [interfaces.py:start():188] Started network monitoring |
| 2025-01-03 20:09:37,329 DEBUG HandlerThread:1963622 [system_info.py:probe():152] Probing system |
| 2025-01-03 20:09:37,330 DEBUG HandlerThread:1963622 [system_info.py:_probe_git():137] Probing git |
| 2025-01-03 20:09:37,335 DEBUG HandlerThread:1963622 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-03 20:09:37,335 DEBUG HandlerThread:1963622 [system_info.py:probe():200] Probing system done |
| 2025-01-03 20:09:37,335 DEBUG HandlerThread:1963622 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-04T01:09:37.329536', 'startedAt': '2025-01-04T01:09:36.985274', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'c537eb6734155592809c2a7db5cca971ec186d72'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3748.4535, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.953, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.998, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 725.9888801574707}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-03 20:09:37,335 INFO HandlerThread:1963622 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-03 20:09:37,335 INFO HandlerThread:1963622 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-03 20:09:37,335 DEBUG HandlerThread:1963622 [system_info.py:_save_code():44] Saving code |
| 2025-01-03 20:09:37,337 DEBUG HandlerThread:1963622 [system_info.py:_save_code():67] Saving code done |
| 2025-01-03 20:09:37,337 DEBUG HandlerThread:1963622 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-03 20:09:37,401 DEBUG HandlerThread:1963622 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-03 20:09:37,401 INFO HandlerThread:1963622 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-03 20:09:37,404 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:09:37,404 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-03 20:09:37,404 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-03 20:09:37,405 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-03 20:09:37,654 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-03 20:09:37,654 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-03 20:09:37,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:09:37,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:09:37,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:37,657 INFO HandlerThread:1963622 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/tb" |
| save: true |
| } |
| control { |
| relay_id: "14d0e77ccdda4224a53b65c76ace9adf" |
| } |
| _info { |
| stream_id: "tmmu86ea" |
| } |
|
|
| 2025-01-03 20:09:37,657 DEBUG HandlerThread:1963622 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-03 20:09:37,694 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/em5prqc5-diff.patch |
| 2025-01-03 20:09:37,722 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/t9vqkeat-code/FISH/eval_robot.py |
| 2025-01-03 20:09:37,904 DEBUG SenderThread:1963622 [sender.py:send():379] send: telemetry |
| 2025-01-03 20:09:37,904 DEBUG SenderThread:1963622 [sender.py:send():379] send: tbrecord |
| 2025-01-03 20:09:37,905 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:09:37,905 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 with policy live |
| 2025-01-03 20:09:37,951 INFO wandb-upload_0:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/dkwn83ij-wandb-metadata.json |
| 2025-01-03 20:09:38,112 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/lhqhujqg-events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:09:38,252 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/code/FISH/eval_robot.py |
| 2025-01-03 20:09:38,252 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:09:38,252 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/diff.patch |
| 2025-01-03 20:09:38,252 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/requirements.txt |
| 2025-01-03 20:09:38,253 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:38,253 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-metadata.json |
| 2025-01-03 20:09:38,253 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/code |
| 2025-01-03 20:09:38,253 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/code/FISH |
| 2025-01-03 20:09:38,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:39,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:40,251 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:40,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:41,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:42,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:09:42,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:43,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:44,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:45,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:46,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:47,072 DEBUG SenderThread:1963622 [sender.py:send():379] send: config |
| 2025-01-03 20:09:47,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:48,080 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:09:48,253 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:48,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:49,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:50,254 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:50,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:51,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:52,255 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:52,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:09:52,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:09:52,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:53,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:53,752 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:09:54,255 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:54,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:55,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:56,256 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:56,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:57,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:58,257 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:09:58,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:09:59,461 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:09:59,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:00,257 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:10:00,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:01,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:02,258 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:10:02,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:03,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:04,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:04,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:05,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:06,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:07,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:10:07,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:10:07,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:08,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:09,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:09,780 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:10,260 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:10:10,260 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/config.yaml |
| 2025-01-03 20:10:10,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:11,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:12,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:13,659 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:14,660 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:15,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:15,660 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:16,660 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:17,660 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:18,660 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:19,660 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:20,661 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:21,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:21,661 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:22,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:10:22,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:10:22,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:23,661 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:24,661 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:25,662 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:26,662 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:26,752 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:27,662 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:28,662 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:29,074 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:10:29,074 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_84a6dabcfe82d3667159.mp4 with policy now |
| 2025-01-03 20:10:29,074 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:10:29,075 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:10:29,075 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:10:29,076 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:10:29,265 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:10:29,265 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/0_eval_0_84a6dabcfe82d3667159.mp4 |
| 2025-01-03 20:10:29,265 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media |
| 2025-01-03 20:10:29,266 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval |
| 2025-01-03 20:10:29,266 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos |
| 2025-01-03 20:10:29,337 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/ab9ne20r-media/videos/eval/0_eval_0_84a6dabcfe82d3667159.mp4 |
| 2025-01-03 20:10:29,598 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/2olv4drx-media/videos/eval/0_eval_0_84a6dabcfe82d3667159.mp4 |
| 2025-01-03 20:10:29,662 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:30,265 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:10:30,663 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:31,663 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:32,083 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:32,663 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:33,663 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:34,267 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:10:34,664 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:35,664 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:36,268 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:10:36,664 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:37,319 DEBUG SystemMonitor:1963622 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-03 20:10:37,320 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:10:37,320 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:37,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:10:37,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:10:37,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:38,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:39,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:40,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:41,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:42,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:42,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:43,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:44,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:45,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:46,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:47,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:48,657 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:48,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:49,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:50,667 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:51,667 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:52,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:10:52,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:10:52,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:53,667 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:53,719 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:54,667 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:55,667 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:56,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:57,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:58,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:10:59,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:10:59,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:00,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:01,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:02,669 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:03,276 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:11:03,564 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/wf7d7cb7-events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:11:03,669 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:04,277 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:04,669 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:05,465 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:05,669 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:06,669 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:07,321 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:11:07,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:11:07,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:11:07,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:08,278 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:08,669 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:09,670 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:10,670 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:10,744 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:11,670 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:12,670 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:13,670 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:14,670 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:15,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:16,663 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:16,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:17,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:18,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:19,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:20,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:21,663 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:21,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:22,655 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:11:22,655 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:11:22,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:23,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:24,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:25,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:26,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:26,728 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:27,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:28,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:29,678 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:30,285 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:30,679 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:31,678 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:32,286 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:32,578 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:32,679 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:33,679 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:34,185 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:11:34,185 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_91c97d4f4d3e6b888a20.mp4 with policy now |
| 2025-01-03 20:11:34,187 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:11:34,188 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:11:34,188 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:11:34,188 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:11:34,286 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:11:34,287 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:34,287 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/1_eval_1_91c97d4f4d3e6b888a20.mp4 |
| 2025-01-03 20:11:34,287 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval |
| 2025-01-03 20:11:34,450 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/vov6gugu-media/videos/eval/1_eval_1_91c97d4f4d3e6b888a20.mp4 |
| 2025-01-03 20:11:34,679 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:34,705 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/zxm8xmsm-media/videos/eval/1_eval_1_91c97d4f4d3e6b888a20.mp4 |
| 2025-01-03 20:11:35,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:36,287 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:36,681 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:37,321 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:11:37,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:11:37,657 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:11:37,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:37,808 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:38,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:39,683 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:40,288 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:40,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:41,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:42,289 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:42,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:43,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:43,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:44,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:45,682 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:46,683 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:47,683 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:48,683 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:49,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:49,684 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:50,683 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:51,684 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:52,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:11:52,657 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:11:52,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:53,685 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:54,293 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:11:54,686 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:55,091 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:11:55,092 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:11:55,092 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:11:55,092 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:11:55,293 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:11:55,686 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:56,294 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:11:56,686 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:57,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:58,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:11:59,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:00,108 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:00,295 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:00,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:01,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:02,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:03,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:04,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:05,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:05,893 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:06,297 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:06,687 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:07,321 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:12:07,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:12:07,657 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:12:07,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:08,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:09,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:10,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:11,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:11,741 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:12,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:13,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:14,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:15,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:16,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:17,666 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:17,689 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:18,689 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:19,689 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:20,689 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:21,689 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:22,656 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:12:22,657 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:12:22,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:22,749 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:23,690 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:24,690 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:25,691 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:26,304 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:26,329 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:12:26,329 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_68dcc088f8980ac14b8c.mp4 with policy now |
| 2025-01-03 20:12:26,332 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:12:26,333 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:12:26,333 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:12:26,334 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:12:26,605 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/w5jvfz8z-media/videos/eval/2_eval_2_68dcc088f8980ac14b8c.mp4 |
| 2025-01-03 20:12:26,691 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:27,304 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:12:27,304 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/2_eval_2_68dcc088f8980ac14b8c.mp4 |
| 2025-01-03 20:12:27,305 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval |
| 2025-01-03 20:12:27,619 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/bekqy5l3-media/videos/eval/2_eval_2_68dcc088f8980ac14b8c.mp4 |
| 2025-01-03 20:12:27,696 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:28,305 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:28,668 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:28,693 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:29,694 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:30,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:31,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:32,306 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:32,698 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:33,697 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:33,885 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:34,307 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:34,699 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:35,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:36,700 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:37,322 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:12:37,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:12:37,658 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:12:37,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:38,700 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:39,167 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:39,308 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:12:39,554 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/e3hgz9mr-events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:12:39,699 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:39,924 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:12:39,925 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:12:39,925 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:12:40,309 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:12:40,309 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:40,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:41,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:42,310 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:42,709 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:43,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:44,311 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:12:44,347 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:44,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:45,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:46,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:47,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:48,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:49,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:49,701 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:50,702 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:51,702 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:52,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:12:52,658 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:12:52,702 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:53,703 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:54,703 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:54,786 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:12:55,703 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:56,703 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:57,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:58,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:12:59,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:00,671 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:00,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:01,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:02,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:03,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:04,704 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:05,705 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:06,672 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:06,705 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:07,322 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:13:07,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:13:07,658 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:13:07,705 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:08,705 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:09,705 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:10,708 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:11,162 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:13:11,162 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_2a78dabf1835350f9eca.mp4 with policy now |
| 2025-01-03 20:13:11,163 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:13:11,163 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:13:11,164 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:13:11,164 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:13:11,320 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:13:11,320 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/3_eval_3_2a78dabf1835350f9eca.mp4 |
| 2025-01-03 20:13:11,321 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval |
| 2025-01-03 20:13:11,432 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/qy9abric-media/videos/eval/3_eval_3_2a78dabf1835350f9eca.mp4 |
| 2025-01-03 20:13:11,691 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/oxql2ic8-media/videos/eval/3_eval_3_2a78dabf1835350f9eca.mp4 |
| 2025-01-03 20:13:11,707 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:12,207 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:12,320 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:12,718 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:13,721 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:14,713 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:15,723 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:16,322 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:16,714 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:17,684 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:17,745 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:18,714 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:19,717 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:20,323 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:20,719 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:21,717 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:22,662 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:13:22,662 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:13:22,719 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:22,757 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:23,716 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:24,723 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:25,716 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:26,722 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:27,721 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:28,688 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:28,721 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:29,729 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:29,984 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:13:29,985 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:13:29,985 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:13:30,326 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:13:30,327 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:13:30,327 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:30,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:31,723 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:32,327 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:32,723 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:33,725 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:34,328 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:34,640 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:34,725 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:35,725 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:36,729 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:37,323 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:13:37,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:13:37,658 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:13:37,725 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:38,725 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:39,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:39,979 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:40,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:41,730 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:42,331 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:42,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:43,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:44,332 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:13:44,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:45,685 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:45,726 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:46,727 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:47,727 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:48,727 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:49,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:50,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:51,685 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:51,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:52,658 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:13:52,659 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:13:52,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:53,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:54,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:55,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:56,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:56,780 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:13:57,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:58,731 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:13:59,743 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:00,741 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:01,514 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:14:01,514 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_7a760bf7d8f336eab9d0.mp4 with policy now |
| 2025-01-03 20:14:01,519 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:14:01,520 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:14:01,520 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:01,520 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:01,745 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:01,773 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/qi1gsn8j-media/videos/eval/4_eval_4_7a760bf7d8f336eab9d0.mp4 |
| 2025-01-03 20:14:02,337 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:02,338 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:02,338 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/4_eval_4_7a760bf7d8f336eab9d0.mp4 |
| 2025-01-03 20:14:02,338 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval |
| 2025-01-03 20:14:02,538 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:14:02,624 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/ta9af08f-media/videos/eval/4_eval_4_7a760bf7d8f336eab9d0.mp4 |
| 2025-01-03 20:14:02,752 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:03,745 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:04,748 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:05,748 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:06,339 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:06,751 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:07,323 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:14:07,665 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:14:07,666 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:14:07,747 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:07,822 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:14:08,749 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:09,751 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:10,340 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:10,765 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:11,755 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:12,036 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:14:12,037 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:12,037 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:12,170 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-03 20:14:12,170 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-03 20:14:12,341 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:12,341 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:14:12,341 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-03 20:14:12,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media |
| 2025-01-03 20:14:12,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table |
| 2025-01-03 20:14:12,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table/eval |
| 2025-01-03 20:14:12,395 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:14:12,554 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/cygzagvo-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-03 20:14:12,585 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/kc2fwgkh-events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:14:12,734 INFO wandb-upload_0:1963622 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp0l23afp7 |
| 2025-01-03 20:14:12,742 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-03 20:14:12,752 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 20:14:12,753 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:13,150 INFO SenderThread:1963622 [sender.py:send_request_log_artifact():1525] logged artifact run-tmmu86ea-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNzgyOTM2NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTY2ODA3MQ==', 'latestArtifact': None}} |
| 2025-01-03 20:14:13,150 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:14:13,150 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json with policy now |
| 2025-01-03 20:14:13,151 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:14:13,151 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:14:13,151 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,152 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,152 DEBUG SenderThread:1963622 [sender.py:send():379] send: telemetry |
| 2025-01-03 20:14:13,152 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,156 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,156 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,156 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,156 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,157 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,157 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,157 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,157 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,158 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,158 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,158 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,158 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,158 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,159 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,159 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,159 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,159 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,160 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,160 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,160 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:13,160 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:13,160 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-03 20:14:13,341 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:13,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:14:13,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table/eval/total_success_rate_ci_6_f1a19274e10d66387b7f.table.json |
| 2025-01-03 20:14:13,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table/eval |
| 2025-01-03 20:14:13,394 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/rcwyk4zs-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-03 20:14:13,535 INFO wandb-upload_0:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/tc56rwhq-media/table/eval/total_success_rate_ci_6_f1a19274e10d66387b7f.table.json |
| 2025-01-03 20:14:13,753 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:14,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:14,479 INFO wandb-upload_1:1963622 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp52e45lg1 |
| 2025-01-03 20:14:14,753 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:14,865 INFO SenderThread:1963622 [sender.py:send_request_log_artifact():1525] logged artifact run-tmmu86ea-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNzgyOTM3OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTY2ODA3NA==', 'latestArtifact': None}} |
| 2025-01-03 20:14:14,865 DEBUG SenderThread:1963622 [sender.py:send():379] send: files |
| 2025-01-03 20:14:14,865 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_f1a19274e10d66387b7f.table.json with policy now |
| 2025-01-03 20:14:14,865 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:14:14,865 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:14,866 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:15,100 INFO wandb-upload_0:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/57s86n7f-media/table/eval/total_success_rate_ci_6_f1a19274e10d66387b7f.table.json |
| 2025-01-03 20:14:15,342 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:15,753 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:16,753 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:16,998 DEBUG SenderThread:1963622 [sender.py:send():379] send: exit |
| 2025-01-03 20:14:16,998 INFO SenderThread:1963622 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-03 20:14:16,998 INFO SenderThread:1963622 [sender.py:send_exit():588] handling runtime: 279 |
| 2025-01-03 20:14:16,999 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:16,999 INFO SenderThread:1963622 [sender.py:send_exit():594] send defer |
| 2025-01-03 20:14:16,999 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:16,999 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-03 20:14:16,999 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:16,999 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-03 20:14:16,999 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-03 20:14:17,000 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:17,000 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-03 20:14:17,000 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:17,000 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-03 20:14:17,000 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-03 20:14:17,000 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:17,000 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-03 20:14:17,000 INFO HandlerThread:1963622 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-03 20:14:17,001 DEBUG SystemMonitor:1963622 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-03 20:14:17,001 DEBUG SystemMonitor:1963622 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-03 20:14:17,001 INFO HandlerThread:1963622 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-03 20:14:17,001 INFO HandlerThread:1963622 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-03 20:14:17,052 INFO HandlerThread:1963622 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-03 20:14:17,052 INFO HandlerThread:1963622 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-03 20:14:17,053 INFO HandlerThread:1963622 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-03 20:14:17,053 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:17,053 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-03 20:14:17,053 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-03 20:14:17,053 DEBUG SenderThread:1963622 [sender.py:send():379] send: stats |
| 2025-01-03 20:14:17,053 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:17,053 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-03 20:14:17,054 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:17,054 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-03 20:14:17,054 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-03 20:14:17,054 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:17,054 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-03 20:14:17,343 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:18,343 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:24,043 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 20:14:24,043 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:14:24,043 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-03 20:14:24,044 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:24,044 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-03 20:14:24,044 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-03 20:14:24,044 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 20:14:24,044 DEBUG SenderThread:1963622 [sender.py:send():379] send: history |
| 2025-01-03 20:14:24,044 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 20:14:24,044 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:24,045 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-03 20:14:24,045 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:24,045 DEBUG SenderThread:1963622 [sender.py:send():379] send: summary |
| 2025-01-03 20:14:24,049 INFO SenderThread:1963622 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 20:14:24,050 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:24,050 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-03 20:14:24,050 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-03 20:14:24,050 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:24,050 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-03 20:14:24,050 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:24,050 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-03 20:14:24,054 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:14:24,163 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-03 20:14:24,163 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:24,163 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-03 20:14:24,163 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:24,163 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-03 20:14:24,279 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-03 20:14:24,279 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:24,279 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-03 20:14:24,279 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:24,279 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-03 20:14:24,279 INFO SenderThread:1963622 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-03 20:14:24,282 INFO SenderThread:1963622 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-03 20:14:24,284 INFO SenderThread:1963622 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-03 20:14:24,293 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-03 20:14:24,293 DEBUG SenderThread:1963622 [sender.py:send():379] send: artifact |
| 2025-01-03 20:14:24,293 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:24,296 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-03 20:14:24,346 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:24,346 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:24,346 INFO Thread-12 :1963622 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/config.yaml |
| 2025-01-03 20:14:24,504 INFO SenderThread:1963622 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNzgyNDA5Mg==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNzgyNDA5Mg==', 'versionIndex': 406}}} |
| 2025-01-03 20:14:24,504 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:24,504 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-03 20:14:24,504 INFO SenderThread:1963622 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-03 20:14:24,999 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 20:14:25,347 INFO SenderThread:1963622 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files |
| 2025-01-03 20:14:25,347 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/config.yaml config.yaml |
| 2025-01-03 20:14:25,347 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json wandb-summary.json |
| 2025-01-03 20:14:25,347 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/requirements.txt requirements.txt |
| 2025-01-03 20:14:25,347 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-03 20:14:25,347 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:14:25,352 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log output.log |
| 2025-01-03 20:14:25,352 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/diff.patch diff.patch |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/1_eval_1_91c97d4f4d3e6b888a20.mp4 media/videos/eval/1_eval_1_91c97d4f4d3e6b888a20.mp4 |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/0_eval_0_84a6dabcfe82d3667159.mp4 media/videos/eval/0_eval_0_84a6dabcfe82d3667159.mp4 |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/3_eval_3_2a78dabf1835350f9eca.mp4 media/videos/eval/3_eval_3_2a78dabf1835350f9eca.mp4 |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/4_eval_4_7a760bf7d8f336eab9d0.mp4 media/videos/eval/4_eval_4_7a760bf7d8f336eab9d0.mp4 |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/videos/eval/2_eval_2_68dcc088f8980ac14b8c.mp4 media/videos/eval/2_eval_2_68dcc088f8980ac14b8c.mp4 |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/media/table/eval/total_success_rate_ci_6_f1a19274e10d66387b7f.table.json media/table/eval/total_success_rate_ci_6_f1a19274e10d66387b7f.table.json |
| 2025-01-03 20:14:25,353 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-03 20:14:25,355 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 20:14:25,356 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:25,356 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-03 20:14:25,356 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:25,357 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-03 20:14:25,357 INFO SenderThread:1963622 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-03 20:14:25,539 INFO wandb-upload_2:1963622 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/requirements.txt |
| 2025-01-03 20:14:25,546 INFO wandb-upload_1:1963622 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/config.yaml |
| 2025-01-03 20:14:25,549 INFO wandb-upload_0:1963622 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/wandb-summary.json |
| 2025-01-03 20:14:25,609 INFO wandb-upload_4:1963622 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/files/output.log |
| 2025-01-03 20:14:25,669 INFO wandb-upload_3:1963622 [upload_job.py:push():130] Uploaded file /tmp/tmppq88rwknwandb/ehqs7jzp-events.out.tfevents.1735952977.leonmkim-ROG-Strix-G15CS-G15CS.1963456.0 |
| 2025-01-03 20:14:25,869 INFO Thread-11 (_thread_body):1963622 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-03 20:14:25,869 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:25,869 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-03 20:14:25,870 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:25,870 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-03 20:14:25,870 INFO SenderThread:1963622 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-03 20:14:25,873 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-03 20:14:25,873 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:25,873 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-03 20:14:25,873 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:25,873 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-03 20:14:25,874 INFO SenderThread:1963622 [file_stream.py:finish():601] file stream finish called |
| 2025-01-03 20:14:25,999 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 20:14:26,080 INFO SenderThread:1963622 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-03 20:14:26,080 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-03 20:14:26,080 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 20:14:26,080 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:26,080 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-03 20:14:26,081 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:26,081 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-03 20:14:26,081 INFO SenderThread:1963622 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-03 20:14:26,081 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 20:14:26,081 INFO HandlerThread:1963622 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-03 20:14:26,081 DEBUG SenderThread:1963622 [sender.py:send():379] send: final |
| 2025-01-03 20:14:26,081 DEBUG SenderThread:1963622 [sender.py:send():379] send: footer |
| 2025-01-03 20:14:26,081 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 20:14:26,081 INFO SenderThread:1963622 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-03 20:14:26,082 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 20:14:26,082 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 20:14:26,082 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-03 20:14:26,082 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 20:14:26,083 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-03 20:14:26,083 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-03 20:14:26,083 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:14:26,084 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: server_info |
| 2025-01-03 20:14:26,138 DEBUG SenderThread:1963622 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 20:14:26,138 INFO MainThread:1963622 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-03 20:14:26,139 INFO MainThread:1963622 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-03 20:14:26,139 INFO MainThread:1963622 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-03 20:14:26,139 DEBUG HandlerThread:1963622 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-03 20:14:26,139 INFO HandlerThread:1963622 [handler.py:finish():882] shutting down handler |
| 2025-01-03 20:14:27,083 INFO WriterThread:1963622 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1938_0/tmmu86ea/wandb/run-20250103_200936-tmmu86ea/run-tmmu86ea.wandb |
| 2025-01-03 20:14:27,138 INFO SenderThread:1963622 [sender.py:finish():1615] shutting down sender |
| 2025-01-03 20:14:27,138 INFO SenderThread:1963622 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-03 20:14:27,138 INFO SenderThread:1963622 [file_pusher.py:join():175] waiting for file pusher |
|
|
