| 2024-12-17 21:06:38,476 INFO StreamThr :73458 [internal.py:wandb_internal():85] W&B internal server running at pid: 73458, started at: 2024-12-17 21:06:38.475833 |
| 2024-12-17 21:06:38,477 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status |
| 2024-12-17 21:06:38,488 INFO WriterThread:73458 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/run-yfogqqas.wandb |
| 2024-12-17 21:06:38,488 DEBUG SenderThread:73458 [sender.py:send():379] send: header |
| 2024-12-17 21:06:38,497 DEBUG SenderThread:73458 [sender.py:send():379] send: run |
| 2024-12-17 21:06:38,810 INFO SenderThread:73458 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files |
| 2024-12-17 21:06:38,810 INFO SenderThread:73458 [sender.py:_start_run_threads():1188] run started: yfogqqas with start time 1734487598.476761 |
| 2024-12-17 21:06:38,824 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-17 21:06:38,825 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: check_version |
| 2024-12-17 21:06:38,874 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-17 21:06:38,878 DEBUG HandlerThread:73458 [system_info.py:__init__():26] System info init |
| 2024-12-17 21:06:38,878 DEBUG HandlerThread:73458 [system_info.py:__init__():41] System info init done |
| 2024-12-17 21:06:38,878 INFO HandlerThread:73458 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-17 21:06:38,879 INFO SystemMonitor:73458 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-17 21:06:38,879 INFO SystemMonitor:73458 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-17 21:06:38,879 INFO HandlerThread:73458 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-17 21:06:38,880 INFO SystemMonitor:73458 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-17 21:06:38,880 INFO SystemMonitor:73458 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-17 21:06:38,881 INFO SystemMonitor:73458 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-17 21:06:38,881 INFO SystemMonitor:73458 [interfaces.py:start():188] Started network monitoring |
| 2024-12-17 21:06:38,892 DEBUG HandlerThread:73458 [system_info.py:probe():152] Probing system |
| 2024-12-17 21:06:38,893 DEBUG HandlerThread:73458 [system_info.py:_probe_git():137] Probing git |
| 2024-12-17 21:06:38,899 DEBUG HandlerThread:73458 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-17 21:06:38,899 DEBUG HandlerThread:73458 [system_info.py:probe():200] Probing system done |
| 2024-12-17 21:06:38,899 DEBUG HandlerThread:73458 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T02:06:38.892197', 'startedAt': '2024-12-18T02:06:38.468085', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3749.986625, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 4499.974, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.984, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 379.4953269958496}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-17 21:06:38,899 INFO HandlerThread:73458 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-17 21:06:38,899 INFO HandlerThread:73458 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-17 21:06:38,899 DEBUG HandlerThread:73458 [system_info.py:_save_code():44] Saving code |
| 2024-12-17 21:06:38,901 DEBUG HandlerThread:73458 [system_info.py:_save_code():67] Saving code done |
| 2024-12-17 21:06:38,901 DEBUG HandlerThread:73458 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-17 21:06:38,967 DEBUG HandlerThread:73458 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-17 21:06:38,967 INFO HandlerThread:73458 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-17 21:06:38,970 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:06:38,970 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-17 21:06:38,971 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-17 21:06:38,971 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-17 21:06:39,216 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-17 21:06:39,216 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-17 21:06:39,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:06:39,218 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:06:39,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:39,220 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/gubu4mfu-diff.patch |
| 2024-12-17 21:06:39,220 INFO HandlerThread:73458 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/tb" |
| save: true |
| } |
| control { |
| relay_id: "49c09d9022f54c26b1b3400b57ff2219" |
| } |
| _info { |
| stream_id: "yfogqqas" |
| } |
|
|
| 2024-12-17 21:06:39,220 DEBUG HandlerThread:73458 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-17 21:06:39,245 INFO wandb-upload_0:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/g7nwelrp-wandb-metadata.json |
| 2024-12-17 21:06:39,272 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/f6xqdkww-code/FISH/eval_robot.py |
| 2024-12-17 21:06:39,304 DEBUG SenderThread:73458 [sender.py:send():379] send: telemetry |
| 2024-12-17 21:06:39,304 DEBUG SenderThread:73458 [sender.py:send():379] send: tbrecord |
| 2024-12-17 21:06:39,304 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:06:39,304 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 with policy live |
| 2024-12-17 21:06:39,497 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/dbc5uhcl-events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:06:39,812 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:39,812 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/code/FISH/eval_robot.py |
| 2024-12-17 21:06:39,812 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:06:39,812 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-metadata.json |
| 2024-12-17 21:06:39,812 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/diff.patch |
| 2024-12-17 21:06:39,813 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/requirements.txt |
| 2024-12-17 21:06:39,813 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/code |
| 2024-12-17 21:06:39,813 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/code/FISH |
| 2024-12-17 21:06:40,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:41,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:41,811 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:42,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:43,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:44,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:06:44,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:44,572 DEBUG SenderThread:73458 [sender.py:send():379] send: config |
| 2024-12-17 21:06:45,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:45,813 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:46,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:47,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:47,813 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:48,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:49,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:06:49,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:49,814 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:50,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:51,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:51,815 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:52,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:53,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:53,815 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:54,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:06:54,217 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:06:54,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:54,347 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:06:55,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:55,816 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:56,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:57,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:57,817 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:06:58,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:59,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:06:59,423 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:06:59,817 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:00,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:01,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:01,818 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:02,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:03,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:03,818 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:04,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:04,521 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:05,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:05,819 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:06,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:07,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:07,820 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:08,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:09,217 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:07:09,218 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:07:09,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:09,535 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:09,820 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:09,820 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/config.yaml |
| 2024-12-17 21:07:10,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:11,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:11,821 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:12,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:13,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:13,822 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:14,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:14,716 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:15,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:15,822 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:16,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:17,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:17,823 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:18,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:19,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:19,721 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:19,823 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:20,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:21,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:21,824 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:22,222 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:23,223 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:23,825 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:24,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:07:24,218 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:07:24,225 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:25,225 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:25,295 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:25,499 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:07:25,499 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_2d8c050f82ae77e13b46.mp4 with policy now |
| 2024-12-17 21:07:25,499 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:07:25,500 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:07:25,500 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:07:25,501 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:07:25,743 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/wlz8556n-media/videos/eval/0_eval_0_2d8c050f82ae77e13b46.mp4 |
| 2024-12-17 21:07:25,825 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:25,826 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/0_eval_0_2d8c050f82ae77e13b46.mp4 |
| 2024-12-17 21:07:25,826 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:07:25,826 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media |
| 2024-12-17 21:07:25,826 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos |
| 2024-12-17 21:07:25,826 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval |
| 2024-12-17 21:07:26,084 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/36pa1z8u-media/videos/eval/0_eval_0_2d8c050f82ae77e13b46.mp4 |
| 2024-12-17 21:07:26,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:27,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:27,826 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:28,225 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:29,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:30,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:31,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:31,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:31,827 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:32,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:33,226 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:33,828 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:34,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:35,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:36,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:37,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:37,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:38,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:38,881 DEBUG SystemMonitor:73458 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-17 21:07:38,882 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:07:39,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:07:39,218 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:07:39,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:39,830 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:07:40,070 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/6g0i9so0-events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:07:40,227 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:41,228 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:41,831 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:42,228 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:42,863 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:43,229 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:43,831 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:44,229 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:45,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:45,832 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:46,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:47,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:47,833 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:47,964 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:48,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:49,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:49,834 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:50,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:51,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:51,834 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:52,230 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:52,968 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:53,231 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:53,835 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:54,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:07:54,218 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:07:54,231 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:55,231 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:55,836 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:56,232 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:57,232 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:57,837 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:07:57,979 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:07:58,234 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:59,232 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:07:59,838 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:00,232 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:01,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:01,839 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:02,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:03,168 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:03,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:03,839 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:04,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:05,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:05,840 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:06,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:07,233 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:07,841 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:08,172 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:08,234 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:08,882 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:08:09,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:08:09,218 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:08:09,234 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:09,842 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:10,234 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:11,023 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:08:11,024 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_ae473354b2f49e42e685.mp4 with policy now |
| 2024-12-17 21:08:11,024 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:08:11,025 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:08:11,025 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:08:11,025 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:08:11,249 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:11,267 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/w7yux075-media/videos/eval/1_eval_1_ae473354b2f49e42e685.mp4 |
| 2024-12-17 21:08:11,842 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:11,843 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:08:11,843 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/1_eval_1_ae473354b2f49e42e685.mp4 |
| 2024-12-17 21:08:11,843 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval |
| 2024-12-17 21:08:12,117 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/211m3p6f-media/videos/eval/1_eval_1_ae473354b2f49e42e685.mp4 |
| 2024-12-17 21:08:12,238 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:13,257 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:13,843 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:14,055 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:14,254 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:15,250 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:15,844 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:16,252 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:17,257 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:18,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:18,845 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:08:19,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:19,244 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:19,598 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:08:19,598 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:08:19,599 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:08:19,845 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:19,845 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:08:20,244 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:21,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:21,846 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:22,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:23,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:23,847 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:24,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:08:24,219 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:08:24,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:24,340 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:25,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:25,847 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:26,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:27,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:27,848 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:28,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:29,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:29,513 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:29,849 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:30,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:31,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:31,849 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:32,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:33,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:33,850 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:34,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:34,712 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:35,246 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:35,851 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:36,247 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:37,247 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:37,852 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:38,247 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:38,883 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:08:39,218 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:08:39,219 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:08:39,247 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:39,852 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:39,905 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:40,247 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:41,249 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:41,853 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:42,248 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:43,248 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:43,854 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:44,248 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:45,102 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:45,249 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:45,854 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:46,249 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:47,249 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:47,855 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:48,249 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:49,250 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:49,856 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:50,149 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:50,258 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:50,629 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:08:50,630 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:08:50,630 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_30bc1b0ae90b1d434cdb.mp4 with policy now |
| 2024-12-17 21:08:50,631 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:08:50,631 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:08:50,631 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:08:50,856 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:08:50,857 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/2_eval_2_30bc1b0ae90b1d434cdb.mp4 |
| 2024-12-17 21:08:50,857 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval |
| 2024-12-17 21:08:50,874 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/o0cx98oz-media/videos/eval/2_eval_2_30bc1b0ae90b1d434cdb.mp4 |
| 2024-12-17 21:08:51,156 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/mpoy6dta-media/videos/eval/2_eval_2_30bc1b0ae90b1d434cdb.mp4 |
| 2024-12-17 21:08:51,255 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:51,857 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:52,253 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:53,259 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:54,221 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:08:54,221 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:08:54,260 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:55,273 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:55,343 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:08:55,858 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:56,260 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:57,258 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:57,859 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:08:58,267 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:59,279 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:08:59,860 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:09:00,097 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/dd7wte82-events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:09:00,280 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:00,437 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:09:00,437 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:09:00,438 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:00,438 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:09:00,860 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:09:01,277 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:01,860 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:02,280 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:03,277 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:04,284 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:05,282 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:05,578 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:05,862 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:06,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:07,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:07,862 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:08,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:08,883 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:09:09,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:09:09,220 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:09:09,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:09,863 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:10,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:10,586 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:11,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:11,864 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:12,283 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:13,284 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:13,865 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:14,284 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:15,284 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:15,594 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:15,865 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:16,284 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:17,284 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:17,866 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:18,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:19,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:19,867 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:20,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:20,785 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:21,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:21,868 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:22,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:23,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:23,870 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:24,219 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:09:24,220 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:09:24,285 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:25,286 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:25,786 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:25,870 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:26,286 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:27,286 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:27,871 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:28,286 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:29,286 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:29,872 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:30,286 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:30,790 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:31,291 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:31,873 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:32,290 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:09:32,290 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:32,290 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_4b8e46037aa4fc1a9999.mp4 with policy now |
| 2024-12-17 21:09:32,293 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:09:32,293 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:09:32,294 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:09:32,294 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:09:32,552 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/ctz0kjak-media/videos/eval/3_eval_3_4b8e46037aa4fc1a9999.mp4 |
| 2024-12-17 21:09:32,873 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:09:32,873 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/3_eval_3_4b8e46037aa4fc1a9999.mp4 |
| 2024-12-17 21:09:32,873 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval |
| 2024-12-17 21:09:33,141 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/xrj0xb74-media/videos/eval/3_eval_3_4b8e46037aa4fc1a9999.mp4 |
| 2024-12-17 21:09:33,295 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:33,873 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:34,320 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:35,316 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:35,837 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:36,297 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:37,325 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:37,875 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:38,293 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:38,884 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:09:39,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:09:39,220 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:09:39,306 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:39,875 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:40,299 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:41,290 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:41,314 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:42,303 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:43,323 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:44,349 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:45,315 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:46,313 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:47,236 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:47,317 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:48,326 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:49,328 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:50,329 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:50,879 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:09:51,322 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:51,498 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:09:51,498 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:09:51,499 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:09:51,879 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:51,879 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:09:52,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:52,809 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:53,326 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:53,880 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:54,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:09:54,220 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:09:54,337 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:55,345 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:55,881 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:56,329 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:57,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:57,881 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:09:57,958 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:09:58,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:59,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:09:59,882 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:00,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:01,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:01,883 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:02,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:02,978 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:03,327 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:03,884 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:04,329 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:05,330 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:05,884 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:06,330 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:07,330 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:07,885 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:07,980 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:08,332 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:08,884 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:10:09,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:10:09,220 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:10:09,330 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:09,886 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:10,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:11,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:11,887 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:12,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:13,179 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:13,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:13,888 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:14,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:15,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:15,888 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:16,331 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:17,332 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:17,889 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:18,198 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:18,332 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:19,332 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:19,890 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:20,332 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:21,332 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:21,891 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:22,333 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:23,333 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:23,382 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:23,891 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:24,220 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:10:24,221 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:10:24,352 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:25,159 DEBUG SenderThread:73458 [sender.py:send():379] send: files |
| 2024-12-17 21:10:25,159 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_724967b55fa1192f5b4e.mp4 with policy now |
| 2024-12-17 21:10:25,170 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:10:25,170 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:10:25,170 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:10:25,171 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:10:25,348 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:25,445 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/t3c4pukr-media/videos/eval/4_eval_4_724967b55fa1192f5b4e.mp4 |
| 2024-12-17 21:10:25,892 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:25,892 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:10:25,892 INFO Thread-12 :73458 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/4_eval_4_724967b55fa1192f5b4e.mp4 |
| 2024-12-17 21:10:25,893 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval |
| 2024-12-17 21:10:26,164 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/139wl2u7-media/videos/eval/4_eval_4_724967b55fa1192f5b4e.mp4 |
| 2024-12-17 21:10:26,345 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:27,362 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:28,356 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:28,736 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:29,358 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:29,893 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:30,357 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:31,359 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:32,367 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:33,367 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:33,896 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:34,245 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:34,383 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:35,374 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:36,372 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:37,371 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:38,376 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:38,885 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:10:39,239 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:10:39,239 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:10:39,314 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:39,398 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:40,391 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:41,404 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:41,556 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:10:41,557 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:10:41,567 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:10:41,898 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:10:41,899 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:10:41,899 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:10:42,127 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /tmp/tmpvo6cq2_2wandb/00wqnzyk-events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:10:42,401 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:43,403 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:44,404 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:45,238 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:45,402 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:46,401 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:47,408 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:48,418 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:49,435 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:50,242 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:50,430 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:51,425 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:52,425 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:53,431 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:54,248 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:10:54,248 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:10:54,430 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:55,388 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:10:55,441 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:56,463 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:57,452 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:10:58,009 DEBUG SenderThread:73458 [sender.py:send():379] send: exit |
| 2024-12-17 21:10:58,009 INFO SenderThread:73458 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-17 21:10:58,010 INFO SenderThread:73458 [sender.py:send_exit():588] handling runtime: 259 |
| 2024-12-17 21:10:58,010 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:10:58,010 INFO SenderThread:73458 [sender.py:send_exit():594] send defer |
| 2024-12-17 21:10:58,010 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:10:58,010 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-17 21:10:58,010 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:10:58,010 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-17 21:10:58,010 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-17 21:10:58,010 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:10:58,010 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-17 21:10:58,011 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:10:58,011 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-17 21:10:58,011 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-17 21:10:58,011 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:10:58,011 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-17 21:10:58,011 INFO HandlerThread:73458 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-17 21:10:58,011 DEBUG SystemMonitor:73458 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-17 21:10:58,011 INFO HandlerThread:73458 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-17 21:10:58,011 DEBUG SystemMonitor:73458 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-17 21:10:58,011 INFO HandlerThread:73458 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-17 21:10:58,060 INFO HandlerThread:73458 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-17 21:10:58,060 INFO HandlerThread:73458 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-17 21:10:58,061 INFO HandlerThread:73458 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-17 21:10:58,061 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:10:58,061 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-17 21:10:58,061 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-17 21:10:58,061 DEBUG SenderThread:73458 [sender.py:send():379] send: stats |
| 2024-12-17 21:10:58,061 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:10:58,061 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-17 21:10:58,061 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:10:58,062 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-17 21:10:58,062 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-17 21:10:58,062 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:10:58,062 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-17 21:10:58,904 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:10:59,904 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:11:04,558 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:11:04,558 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:11:04,559 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:04,559 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-17 21:11:04,559 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-17 21:11:04,559 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-17 21:11:04,559 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:11:04,559 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:04,559 DEBUG SenderThread:73458 [sender.py:send():379] send: history |
| 2024-12-17 21:11:04,560 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-17 21:11:04,560 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:11:04,561 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:11:04,561 DEBUG SenderThread:73458 [sender.py:send():379] send: summary |
| 2024-12-17 21:11:04,565 INFO SenderThread:73458 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:11:04,565 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:04,565 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-17 21:11:04,565 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-17 21:11:04,565 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:04,565 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-17 21:11:04,565 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:04,566 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-17 21:11:04,566 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-17 21:11:04,566 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:11:04,566 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:04,566 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-17 21:11:04,566 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:04,566 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-17 21:11:04,906 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:11:05,010 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:11:07,129 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-17 21:11:07,129 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:11:07,129 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:07,129 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-17 21:11:07,130 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:07,130 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-17 21:11:07,130 INFO SenderThread:73458 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-17 21:11:07,132 INFO SenderThread:73458 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-17 21:11:07,152 INFO SenderThread:73458 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-17 21:11:07,161 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-17 21:11:07,161 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:07,162 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-17 21:11:07,162 DEBUG SenderThread:73458 [sender.py:send():379] send: artifact |
| 2024-12-17 21:11:07,590 INFO wandb-upload_0:73458 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp3z3b2uta |
| 2024-12-17 21:11:07,590 INFO wandb-upload_1:73458 [upload_job.py:push():85] Skipped uploading /tmp/tmpwx7g_z5t/wandb-job.json |
| 2024-12-17 21:11:07,692 INFO wandb-upload_2:73458 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmppfin8p16 |
| 2024-12-17 21:11:07,907 INFO Thread-12 :73458 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:11:08,012 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:11:08,290 INFO SenderThread:73458 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzE0NTc2Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4MzExNzUzMw==', 'versionIndex': 376}}} |
| 2024-12-17 21:11:08,290 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:08,290 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-17 21:11:08,291 INFO SenderThread:73458 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-17 21:11:08,908 INFO SenderThread:73458 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files |
| 2024-12-17 21:11:08,908 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/config.yaml config.yaml |
| 2024-12-17 21:11:08,908 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 events.out.tfevents.1734487599.leonmkim-ROG-Strix-G15CS-G15CS.73269.0 |
| 2024-12-17 21:11:08,908 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json wandb-summary.json |
| 2024-12-17 21:11:08,910 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/requirements.txt requirements.txt |
| 2024-12-17 21:11:08,910 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-17 21:11:08,912 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log output.log |
| 2024-12-17 21:11:08,913 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/diff.patch diff.patch |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/3_eval_3_4b8e46037aa4fc1a9999.mp4 media/videos/eval/3_eval_3_4b8e46037aa4fc1a9999.mp4 |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/2_eval_2_30bc1b0ae90b1d434cdb.mp4 media/videos/eval/2_eval_2_30bc1b0ae90b1d434cdb.mp4 |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/4_eval_4_724967b55fa1192f5b4e.mp4 media/videos/eval/4_eval_4_724967b55fa1192f5b4e.mp4 |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/0_eval_0_2d8c050f82ae77e13b46.mp4 media/videos/eval/0_eval_0_2d8c050f82ae77e13b46.mp4 |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/media/videos/eval/1_eval_1_ae473354b2f49e42e685.mp4 media/videos/eval/1_eval_1_ae473354b2f49e42e685.mp4 |
| 2024-12-17 21:11:08,915 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-17 21:11:08,915 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:11:08,915 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:08,916 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-17 21:11:08,916 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:08,916 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-17 21:11:08,916 INFO SenderThread:73458 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-17 21:11:09,012 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:11:09,012 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:11:09,088 INFO wandb-upload_2:73458 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/wandb-summary.json |
| 2024-12-17 21:11:09,110 INFO wandb-upload_0:73458 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/config.yaml |
| 2024-12-17 21:11:09,165 INFO wandb-upload_1:73458 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/requirements.txt |
| 2024-12-17 21:11:09,191 INFO wandb-upload_3:73458 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/files/output.log |
| 2024-12-17 21:11:09,391 INFO Thread-11 (_thread_body):73458 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-17 21:11:09,392 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:09,392 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-17 21:11:09,392 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:09,392 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-17 21:11:09,392 INFO SenderThread:73458 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-17 21:11:09,393 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-17 21:11:09,393 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:09,393 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-17 21:11:09,393 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:09,393 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-17 21:11:09,393 INFO SenderThread:73458 [file_stream.py:finish():601] file stream finish called |
| 2024-12-17 21:11:09,575 INFO SenderThread:73458 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-17 21:11:09,575 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-17 21:11:09,575 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:09,575 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-17 21:11:09,576 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:09,576 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-17 21:11:09,576 INFO SenderThread:73458 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-17 21:11:09,576 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:11:09,576 DEBUG SenderThread:73458 [sender.py:send():379] send: final |
| 2024-12-17 21:11:09,576 INFO HandlerThread:73458 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-17 21:11:09,576 DEBUG SenderThread:73458 [sender.py:send():379] send: footer |
| 2024-12-17 21:11:09,576 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:11:09,576 INFO SenderThread:73458 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-17 21:11:09,577 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:11:09,577 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:11:09,577 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-17 21:11:09,577 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:11:09,577 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: server_info |
| 2024-12-17 21:11:09,579 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-17 21:11:09,579 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-17 21:11:09,579 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:11:09,644 DEBUG SenderThread:73458 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:11:09,644 INFO MainThread:73458 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-17 21:11:09,644 INFO MainThread:73458 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-17 21:11:09,645 INFO MainThread:73458 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-17 21:11:09,645 DEBUG HandlerThread:73458 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-17 21:11:09,645 INFO HandlerThread:73458 [handler.py:finish():882] shutting down handler |
| 2024-12-17 21:11:10,578 INFO WriterThread:73458 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/522_0/210632/wandb/run-20241217_210638-yfogqqas/run-yfogqqas.wandb |
| 2024-12-17 21:11:10,644 INFO SenderThread:73458 [sender.py:finish():1615] shutting down sender |
| 2024-12-17 21:11:10,644 INFO SenderThread:73458 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-17 21:11:10,644 INFO SenderThread:73458 [file_pusher.py:join():175] waiting for file pusher |
|
|
