| 2024-12-31 21:14:28,570 INFO StreamThr :1742514 [internal.py:wandb_internal():85] W&B internal server running at pid: 1742514, started at: 2024-12-31 21:14:28.569821 |
| 2024-12-31 21:14:28,571 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status |
| 2024-12-31 21:14:28,581 INFO WriterThread:1742514 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/run-3o6eq09y.wandb |
| 2024-12-31 21:14:28,582 DEBUG SenderThread:1742514 [sender.py:send():379] send: header |
| 2024-12-31 21:14:28,591 DEBUG SenderThread:1742514 [sender.py:send():379] send: run |
| 2024-12-31 21:14:28,822 INFO SenderThread:1742514 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files |
| 2024-12-31 21:14:28,822 INFO SenderThread:1742514 [sender.py:_start_run_threads():1188] run started: 3o6eq09y with start time 1735697668.570133 |
| 2024-12-31 21:14:28,841 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-31 21:14:28,841 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: check_version |
| 2024-12-31 21:14:28,889 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-31 21:14:28,894 DEBUG HandlerThread:1742514 [system_info.py:__init__():26] System info init |
| 2024-12-31 21:14:28,894 DEBUG HandlerThread:1742514 [system_info.py:__init__():41] System info init done |
| 2024-12-31 21:14:28,894 INFO HandlerThread:1742514 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-31 21:14:28,894 INFO SystemMonitor:1742514 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-31 21:14:28,894 INFO HandlerThread:1742514 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-31 21:14:28,895 INFO SystemMonitor:1742514 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-31 21:14:28,895 INFO SystemMonitor:1742514 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-31 21:14:28,896 INFO SystemMonitor:1742514 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-31 21:14:28,897 INFO SystemMonitor:1742514 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-31 21:14:28,898 INFO SystemMonitor:1742514 [interfaces.py:start():188] Started network monitoring |
| 2024-12-31 21:14:28,907 DEBUG HandlerThread:1742514 [system_info.py:probe():152] Probing system |
| 2024-12-31 21:14:28,909 DEBUG HandlerThread:1742514 [system_info.py:_probe_git():137] Probing git |
| 2024-12-31 21:14:28,913 DEBUG HandlerThread:1742514 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-31 21:14:28,913 DEBUG HandlerThread:1742514 [system_info.py:probe():200] Probing system done |
| 2024-12-31 21:14:28,913 DEBUG HandlerThread:1742514 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T02:14:28.907694', 'startedAt': '2025-01-01T02:14:28.560566', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '284407f87cd673789a10dbd8e4d4f41dcad30ded'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.9964999999997, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4500.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.913, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 500.70996856689453}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-31 21:14:28,913 INFO HandlerThread:1742514 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-31 21:14:28,914 INFO HandlerThread:1742514 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-31 21:14:28,914 DEBUG HandlerThread:1742514 [system_info.py:_save_code():44] Saving code |
| 2024-12-31 21:14:28,919 DEBUG HandlerThread:1742514 [system_info.py:_save_code():67] Saving code done |
| 2024-12-31 21:14:28,919 DEBUG HandlerThread:1742514 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-31 21:14:28,981 DEBUG HandlerThread:1742514 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-31 21:14:28,981 INFO HandlerThread:1742514 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-31 21:14:28,984 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:14:28,984 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-31 21:14:28,984 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-31 21:14:28,985 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-31 21:14:29,216 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-31 21:14:29,216 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-31 21:14:29,218 DEBUG SenderThread:1742514 [sender.py:send():379] send: telemetry |
| 2024-12-31 21:14:29,249 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/guxb7ia0-wandb-metadata.json |
| 2024-12-31 21:14:29,255 INFO wandb-upload_1:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/9e4m240a-code/FISH/eval_robot.py |
| 2024-12-31 21:14:29,261 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:14:29,261 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:29,261 INFO HandlerThread:1742514 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/tb" |
| save: true |
| } |
| control { |
| relay_id: "aa99e0ae8fd2488f967e68a770b7a1fa" |
| } |
| _info { |
| stream_id: "3o6eq09y" |
| } |
|
|
| 2024-12-31 21:14:29,261 DEBUG HandlerThread:1742514 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-31 21:14:29,262 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:14:29,295 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/u5obav2n-diff.patch |
| 2024-12-31 21:14:29,420 DEBUG SenderThread:1742514 [sender.py:send():379] send: tbrecord |
| 2024-12-31 21:14:29,421 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:14:29,421 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 with policy live |
| 2024-12-31 21:14:29,613 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/wmnda6o5-events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/diff.patch |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/code/FISH/eval_robot.py |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/requirements.txt |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-metadata.json |
| 2024-12-31 21:14:29,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/code |
| 2024-12-31 21:14:29,825 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/code/FISH |
| 2024-12-31 21:14:30,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:31,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:31,823 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:32,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:33,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:34,072 DEBUG SenderThread:1742514 [sender.py:send():379] send: config |
| 2024-12-31 21:14:34,072 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:14:34,218 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:35,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:35,824 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:36,218 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:37,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:37,825 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:38,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:39,218 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:14:39,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:39,825 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:40,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:41,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:42,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:43,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:43,826 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:44,216 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:14:44,217 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:14:44,257 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:44,306 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:14:45,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:46,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:47,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:48,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:49,221 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:50,218 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:14:50,221 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:51,221 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:52,221 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:53,221 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:53,829 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:54,221 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:55,222 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:55,276 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:14:56,222 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:57,222 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:57,830 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:14:58,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:59,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:14:59,217 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:14:59,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:14:59,831 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:00,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:00,419 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:00,831 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/config.yaml |
| 2024-12-31 21:15:01,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:01,831 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:02,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:03,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:03,832 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:04,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:05,224 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:05,615 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:05,833 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:06,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:07,224 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:07,833 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:08,224 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:09,224 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:09,834 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:10,224 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:10,808 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:11,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:11,834 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:12,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:13,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:13,835 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:14,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:15:14,217 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:15:14,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:15,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:15,811 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:15,835 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:16,225 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:17,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:17,836 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:18,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:19,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:19,837 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:20,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:20,821 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:21,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:21,837 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:22,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:23,109 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:15:23,109 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_4a716cd980b5c787ae82.mp4 with policy now |
| 2024-12-31 21:15:23,110 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:15:23,110 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:15:23,111 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:15:23,111 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:15:23,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:23,351 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/i0m5psba-media/videos/eval/0_eval_0_4a716cd980b5c787ae82.mp4 |
| 2024-12-31 21:15:23,838 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:23,838 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/0_eval_0_4a716cd980b5c787ae82.mp4 |
| 2024-12-31 21:15:23,838 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:15:23,838 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval |
| 2024-12-31 21:15:23,838 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos |
| 2024-12-31 21:15:23,838 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media |
| 2024-12-31 21:15:24,121 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/anzuiri3-media/videos/eval/0_eval_0_4a716cd980b5c787ae82.mp4 |
| 2024-12-31 21:15:24,227 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:25,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:25,839 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:26,219 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:26,232 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:27,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:27,839 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:28,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:28,898 DEBUG SystemMonitor:1742514 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-31 21:15:28,899 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:15:29,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:15:29,217 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:15:29,265 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:30,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:31,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:31,485 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:31,841 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:32,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:33,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:34,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:35,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:36,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:37,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:37,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:38,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:39,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:40,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:41,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:42,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:42,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:43,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:43,844 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:15:43,845 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:44,088 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/cospvvhe-events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:15:44,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:15:44,217 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:15:44,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:45,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:45,845 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:46,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:47,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:47,417 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:47,846 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:48,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:49,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:49,847 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:50,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:51,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:51,847 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:52,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:52,441 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:53,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:53,848 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:54,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:55,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:55,849 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:56,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:57,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:57,444 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:15:57,850 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:15:58,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:58,899 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:15:59,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:15:59,217 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:15:59,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:15:59,850 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:00,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:01,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:01,756 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:16:01,756 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_7a4613e2e9f3592faf3d.mp4 with policy now |
| 2024-12-31 21:16:01,756 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:16:01,757 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:16:01,757 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:16:01,757 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:16:01,851 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:01,851 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:16:01,851 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/1_eval_1_7a4613e2e9f3592faf3d.mp4 |
| 2024-12-31 21:16:01,851 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval |
| 2024-12-31 21:16:01,968 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/wd6qovxs-media/videos/eval/1_eval_1_7a4613e2e9f3592faf3d.mp4 |
| 2024-12-31 21:16:02,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:02,253 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/yi05qoxg-media/videos/eval/1_eval_1_7a4613e2e9f3592faf3d.mp4 |
| 2024-12-31 21:16:02,782 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:03,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:03,852 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:04,233 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:05,234 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:06,242 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:07,234 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:07,853 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:08,077 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:08,233 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:09,233 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:09,854 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:10,233 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:11,233 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:12,234 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:13,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:13,234 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:14,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:16:14,218 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:16:14,261 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:15,234 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:16,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:17,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:18,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:18,375 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:19,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:20,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:21,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:21,664 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:16:21,665 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:16:21,665 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:16:21,858 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:16:21,858 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:21,858 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:16:22,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:23,235 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:23,859 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:24,089 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:24,236 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:25,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:25,859 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:26,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:27,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:27,860 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:28,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:28,899 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:16:29,151 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:29,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:16:29,218 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:16:29,238 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:29,861 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:30,238 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:31,239 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:31,861 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:32,239 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:33,239 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:33,862 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:34,223 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:34,239 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:35,239 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:35,863 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:36,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:37,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:37,864 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:38,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:39,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:39,339 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:39,864 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:40,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:41,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:41,865 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:42,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:43,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:43,866 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:44,217 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:16:44,218 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:16:44,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:44,365 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:45,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:45,867 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:46,240 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:47,241 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:47,867 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:48,241 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:49,243 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:49,553 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:49,868 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:50,242 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:51,243 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:51,869 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:52,244 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:53,248 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:53,419 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:16:53,420 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_9e34a800c620d3c2e83a.mp4 with policy now |
| 2024-12-31 21:16:53,422 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:16:53,423 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:16:53,423 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:16:53,423 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:16:53,681 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/0iq7tr7l-media/videos/eval/2_eval_2_9e34a800c620d3c2e83a.mp4 |
| 2024-12-31 21:16:53,869 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:53,870 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:16:53,870 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/2_eval_2_9e34a800c620d3c2e83a.mp4 |
| 2024-12-31 21:16:53,870 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval |
| 2024-12-31 21:16:54,147 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/mujn882g-media/videos/eval/2_eval_2_9e34a800c620d3c2e83a.mp4 |
| 2024-12-31 21:16:54,253 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:55,237 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:16:55,255 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:55,870 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:56,257 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:57,253 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:57,871 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:16:58,253 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:16:58,900 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:16:59,242 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:16:59,242 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:16:59,285 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:00,252 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:00,957 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:01,256 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:01,872 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:02,262 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:03,257 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:04,258 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:05,256 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:06,238 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:06,257 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:07,264 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:08,262 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:09,263 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:10,264 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:11,272 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:12,231 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:12,265 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:13,261 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:14,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:17:14,230 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:17:14,262 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:14,510 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:17:14,510 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:17:14,511 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:17:14,876 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:17:14,877 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:17:15,160 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/2k8lxt88-events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:17:15,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:15,877 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:16,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:17,275 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:17,857 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:18,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:19,267 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:19,878 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:20,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:21,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:21,879 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:22,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:23,030 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:23,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:23,880 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:24,269 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:25,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:25,882 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:26,268 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:27,269 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:27,883 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:28,222 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:28,269 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:28,900 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:17:29,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:17:29,227 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:17:29,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:29,883 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:30,269 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:31,269 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:31,884 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:32,269 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:33,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:33,416 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:33,885 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:34,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:35,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:35,886 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:36,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:37,271 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:37,887 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:38,270 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:38,453 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:39,289 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:39,887 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:40,272 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:41,272 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:41,888 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:42,272 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:43,272 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:43,615 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:43,889 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:44,226 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:17:44,227 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:17:44,273 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:45,272 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:45,890 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:46,280 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:46,322 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:17:46,322 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_87f3ae88ae1b7eb12483.mp4 with policy now |
| 2024-12-31 21:17:46,365 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:17:46,366 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:17:46,366 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:17:46,366 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:17:46,569 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/lltkthey-media/videos/eval/3_eval_3_87f3ae88ae1b7eb12483.mp4 |
| 2024-12-31 21:17:46,890 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:17:46,890 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/3_eval_3_87f3ae88ae1b7eb12483.mp4 |
| 2024-12-31 21:17:46,891 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval |
| 2024-12-31 21:17:47,195 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/t7iqs0vg-media/videos/eval/3_eval_3_87f3ae88ae1b7eb12483.mp4 |
| 2024-12-31 21:17:47,281 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:47,891 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:48,275 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:49,233 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:49,284 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:50,294 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:51,293 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:51,892 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:52,293 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:53,280 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:53,893 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:54,287 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:55,232 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:17:55,296 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:55,562 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:17:55,562 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:17:55,562 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:17:55,893 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:17:55,894 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:55,894 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:17:56,295 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:57,299 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:57,894 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:17:58,299 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:58,901 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:17:59,229 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:17:59,229 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:17:59,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:17:59,895 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:00,304 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:00,318 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:01,296 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:01,896 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:02,296 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:03,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:03,896 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:04,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:05,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:05,513 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:05,897 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:06,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:07,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:07,898 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:08,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:09,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:09,899 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:10,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:10,712 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:11,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:11,900 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:12,297 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:13,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:13,900 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:14,227 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:18:14,227 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:18:14,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:15,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:15,714 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:15,901 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:16,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:17,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:17,902 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:18,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:19,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:19,906 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:20,298 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:20,722 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:21,299 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:22,301 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:22,907 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:23,301 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:24,301 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:24,908 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:25,301 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:25,912 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:26,302 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:26,909 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:27,302 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:28,426 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:28,901 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:18:28,910 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:29,214 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:18:29,214 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_1fea62f600c288e0eddf.mp4 with policy now |
| 2024-12-31 21:18:29,220 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:18:29,221 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:18:29,221 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:29,221 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:29,230 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:18:29,230 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:18:29,411 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:29,443 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/8ei8ufpm-media/videos/eval/4_eval_4_1fea62f600c288e0eddf.mp4 |
| 2024-12-31 21:18:29,910 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:29,910 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/4_eval_4_1fea62f600c288e0eddf.mp4 |
| 2024-12-31 21:18:29,911 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval |
| 2024-12-31 21:18:30,213 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/vw2dpf8c-media/videos/eval/4_eval_4_1fea62f600c288e0eddf.mp4 |
| 2024-12-31 21:18:30,413 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:30,911 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:31,387 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:31,413 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:32,419 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:33,427 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:34,427 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:34,912 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:35,429 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:36,429 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:36,769 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:37,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:38,433 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:38,913 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:39,435 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:40,443 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:40,624 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:18:40,624 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:40,625 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:40,914 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:18:40,914 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:41,002 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 21:18:41,002 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 21:18:41,159 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/1rmx3nu1-events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:18:41,181 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:18:41,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:41,458 INFO wandb-upload_0:1742514 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpb0c3kog2 |
| 2024-12-31 21:18:41,595 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 21:18:41,605 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 21:18:41,860 INFO SenderThread:1742514 [sender.py:send_request_log_artifact():1525] logged artifact run-3o6eq09y-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDEyMTQyOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE2NzUwMQ==', 'latestArtifact': None}} |
| 2024-12-31 21:18:41,860 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:18:41,860 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json with policy now |
| 2024-12-31 21:18:41,860 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:18:41,860 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,861 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:41,861 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,861 DEBUG SenderThread:1742514 [sender.py:send():379] send: telemetry |
| 2024-12-31 21:18:41,861 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,862 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,862 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,862 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,862 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,864 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,864 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,865 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,865 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,865 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,865 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,865 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,866 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,866 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,866 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,866 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,867 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,867 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,867 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,867 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,867 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:41,868 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:41,868 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 21:18:41,914 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:18:41,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:41,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/table/eval/total_success_rate_ci_6_53016dfb1bc53388f97a.table.json |
| 2024-12-31 21:18:41,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2024-12-31 21:18:41,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media |
| 2024-12-31 21:18:41,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/table |
| 2024-12-31 21:18:41,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/table/eval |
| 2024-12-31 21:18:42,133 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/1kmzoaem-media/table/eval/total_success_rate_ci_6_53016dfb1bc53388f97a.table.json |
| 2024-12-31 21:18:42,153 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/4ee8579n-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2024-12-31 21:18:42,255 INFO wandb-upload_1:1742514 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpp7x4evu9 |
| 2024-12-31 21:18:42,402 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/oian0evt-media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2024-12-31 21:18:42,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:42,644 INFO SenderThread:1742514 [sender.py:send_request_log_artifact():1525] logged artifact run-3o6eq09y-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDEyMTQzMw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE2NzUwMw==', 'latestArtifact': None}} |
| 2024-12-31 21:18:42,644 DEBUG SenderThread:1742514 [sender.py:send():379] send: files |
| 2024-12-31 21:18:42,644 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_53016dfb1bc53388f97a.table.json with policy now |
| 2024-12-31 21:18:42,644 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:18:42,644 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:42,645 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:42,889 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/xvyvn711-media/table/eval/total_success_rate_ci_6_53016dfb1bc53388f97a.table.json |
| 2024-12-31 21:18:42,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:42,915 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:43,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:44,228 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 21:18:44,228 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 21:18:44,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:44,916 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:45,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:46,432 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:47,293 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:47,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:48,431 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:49,410 DEBUG SenderThread:1742514 [sender.py:send():379] send: exit |
| 2024-12-31 21:18:49,410 INFO SenderThread:1742514 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-31 21:18:49,410 INFO SenderThread:1742514 [sender.py:send_exit():588] handling runtime: 260 |
| 2024-12-31 21:18:49,411 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:49,411 INFO SenderThread:1742514 [sender.py:send_exit():594] send defer |
| 2024-12-31 21:18:49,411 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:49,411 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-31 21:18:49,412 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:49,412 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-31 21:18:49,412 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-31 21:18:49,412 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:49,412 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-31 21:18:49,412 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:49,412 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-31 21:18:49,412 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-31 21:18:49,412 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:49,412 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-31 21:18:49,412 INFO HandlerThread:1742514 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-31 21:18:49,412 DEBUG SystemMonitor:1742514 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-31 21:18:49,413 DEBUG SystemMonitor:1742514 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-31 21:18:49,413 INFO HandlerThread:1742514 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-31 21:18:49,413 INFO HandlerThread:1742514 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-31 21:18:49,466 INFO HandlerThread:1742514 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-31 21:18:49,466 INFO HandlerThread:1742514 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-31 21:18:49,467 INFO HandlerThread:1742514 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-31 21:18:49,467 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:49,467 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-31 21:18:49,467 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-31 21:18:49,467 DEBUG SenderThread:1742514 [sender.py:send():379] send: stats |
| 2024-12-31 21:18:49,467 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:49,467 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-31 21:18:49,468 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:49,468 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-31 21:18:49,468 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-31 21:18:49,468 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:49,468 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-31 21:18:49,918 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:50,918 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:55,630 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 21:18:55,630 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:55,630 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-31 21:18:55,631 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:55,631 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-31 21:18:55,631 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-31 21:18:55,631 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 21:18:55,631 DEBUG SenderThread:1742514 [sender.py:send():379] send: history |
| 2024-12-31 21:18:55,631 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:55,632 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-31 21:18:55,631 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 21:18:55,632 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:55,633 DEBUG SenderThread:1742514 [sender.py:send():379] send: summary |
| 2024-12-31 21:18:55,633 INFO SenderThread:1742514 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 21:18:55,633 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:55,633 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-31 21:18:55,633 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-31 21:18:55,633 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:55,633 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-31 21:18:55,633 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:55,633 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-31 21:18:55,637 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 21:18:55,783 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-31 21:18:55,783 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:55,783 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-31 21:18:55,783 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:55,783 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-31 21:18:55,920 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:55,921 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/config.yaml |
| 2024-12-31 21:18:56,410 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 21:18:57,477 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-31 21:18:57,477 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 21:18:57,477 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:57,477 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-31 21:18:57,477 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:57,478 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-31 21:18:57,478 INFO SenderThread:1742514 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-31 21:18:57,480 INFO SenderThread:1742514 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-31 21:18:57,482 INFO SenderThread:1742514 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-31 21:18:57,496 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-31 21:18:57,496 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:57,497 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-31 21:18:57,497 DEBUG SenderThread:1742514 [sender.py:send():379] send: artifact |
| 2024-12-31 21:18:57,881 INFO wandb-upload_1:1742514 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpdc1ni3pt |
| 2024-12-31 21:18:57,921 INFO Thread-12 :1742514 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:58,011 INFO wandb-upload_0:1742514 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppld45pz6 |
| 2024-12-31 21:18:58,064 INFO wandb-upload_2:1742514 [upload_job.py:push():88] Uploaded file /tmp/tmp5hfd5tbw/wandb-job.json |
| 2024-12-31 21:18:58,411 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 21:18:58,692 INFO SenderThread:1742514 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDEyMTU3MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDExODg5MA==', 'versionIndex': 391}}} |
| 2024-12-31 21:18:58,692 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:58,692 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-31 21:18:58,692 INFO SenderThread:1742514 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-31 21:18:58,922 INFO SenderThread:1742514 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files |
| 2024-12-31 21:18:58,922 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/config.yaml config.yaml |
| 2024-12-31 21:18:58,922 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json wandb-summary.json |
| 2024-12-31 21:18:58,922 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:18:58,924 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/requirements.txt requirements.txt |
| 2024-12-31 21:18:58,925 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-31 21:18:58,925 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log output.log |
| 2024-12-31 21:18:58,930 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/diff.patch diff.patch |
| 2024-12-31 21:18:58,931 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-31 21:18:58,931 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/3_eval_3_87f3ae88ae1b7eb12483.mp4 media/videos/eval/3_eval_3_87f3ae88ae1b7eb12483.mp4 |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/1_eval_1_7a4613e2e9f3592faf3d.mp4 media/videos/eval/1_eval_1_7a4613e2e9f3592faf3d.mp4 |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/2_eval_2_9e34a800c620d3c2e83a.mp4 media/videos/eval/2_eval_2_9e34a800c620d3c2e83a.mp4 |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/0_eval_0_4a716cd980b5c787ae82.mp4 media/videos/eval/0_eval_0_4a716cd980b5c787ae82.mp4 |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/videos/eval/4_eval_4_1fea62f600c288e0eddf.mp4 media/videos/eval/4_eval_4_1fea62f600c288e0eddf.mp4 |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json media/table/eval/success_rate_ci_5_f6f07faa4a586efd9baa.table.json |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/media/table/eval/total_success_rate_ci_6_53016dfb1bc53388f97a.table.json media/table/eval/total_success_rate_ci_6_53016dfb1bc53388f97a.table.json |
| 2024-12-31 21:18:58,932 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-31 21:18:58,932 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 21:18:58,932 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:58,932 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-31 21:18:58,933 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:58,933 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-31 21:18:58,933 INFO SenderThread:1742514 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 21:18:59,107 INFO wandb-upload_1:1742514 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/config.yaml |
| 2024-12-31 21:18:59,138 INFO wandb-upload_0:1742514 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/wandb-summary.json |
| 2024-12-31 21:18:59,148 INFO wandb-upload_2:1742514 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/requirements.txt |
| 2024-12-31 21:18:59,163 INFO wandb-upload_3:1742514 [upload_job.py:push():130] Uploaded file /tmp/tmpjwrk1enowandb/r1mgel0l-events.out.tfevents.1735697669.leonmkim-ROG-Strix-G15CS-G15CS.1742349.0 |
| 2024-12-31 21:18:59,164 INFO wandb-upload_4:1742514 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/files/output.log |
| 2024-12-31 21:18:59,364 INFO Thread-11 (_thread_body):1742514 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-31 21:18:59,364 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:59,365 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-31 21:18:59,365 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:59,365 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-31 21:18:59,365 INFO SenderThread:1742514 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-31 21:18:59,366 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-31 21:18:59,366 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:59,366 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-31 21:18:59,366 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:59,366 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-31 21:18:59,367 INFO SenderThread:1742514 [file_stream.py:finish():601] file stream finish called |
| 2024-12-31 21:18:59,411 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 21:18:59,572 INFO SenderThread:1742514 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-31 21:18:59,572 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-31 21:18:59,572 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 21:18:59,572 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:59,572 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-31 21:18:59,573 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:59,573 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-31 21:18:59,573 INFO SenderThread:1742514 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-31 21:18:59,573 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 21:18:59,573 INFO HandlerThread:1742514 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-31 21:18:59,573 DEBUG SenderThread:1742514 [sender.py:send():379] send: final |
| 2024-12-31 21:18:59,573 DEBUG SenderThread:1742514 [sender.py:send():379] send: footer |
| 2024-12-31 21:18:59,573 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 21:18:59,573 INFO SenderThread:1742514 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-31 21:18:59,574 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 21:18:59,574 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-31 21:18:59,574 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 21:18:59,574 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: server_info |
| 2024-12-31 21:18:59,576 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 21:18:59,576 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-31 21:18:59,577 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-31 21:18:59,577 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 21:18:59,626 DEBUG SenderThread:1742514 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 21:18:59,627 INFO MainThread:1742514 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-31 21:18:59,627 INFO MainThread:1742514 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-31 21:18:59,627 INFO MainThread:1742514 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-31 21:18:59,628 DEBUG HandlerThread:1742514 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-31 21:18:59,628 INFO HandlerThread:1742514 [handler.py:finish():882] shutting down handler |
| 2024-12-31 21:19:00,576 INFO WriterThread:1742514 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/3o6eq09y/wandb/run-20241231_211428-3o6eq09y/run-3o6eq09y.wandb |
| 2024-12-31 21:19:00,627 INFO SenderThread:1742514 [sender.py:finish():1615] shutting down sender |
| 2024-12-31 21:19:00,627 INFO SenderThread:1742514 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 21:19:00,627 INFO SenderThread:1742514 [file_pusher.py:join():175] waiting for file pusher |
|
|
