| 2024-12-10 21:04:14,256 INFO StreamThr :600018 [internal.py:wandb_internal():85] W&B internal server running at pid: 600018, started at: 2024-12-10 21:04:14.256478 |
| 2024-12-10 21:04:14,265 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status |
| 2024-12-10 21:04:14,269 INFO WriterThread:600018 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/run-ob7chcle.wandb |
| 2024-12-10 21:04:14,270 DEBUG SenderThread:600018 [sender.py:send():379] send: header |
| 2024-12-10 21:04:14,286 DEBUG SenderThread:600018 [sender.py:send():379] send: run |
| 2024-12-10 21:04:14,541 INFO SenderThread:600018 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files |
| 2024-12-10 21:04:14,541 INFO SenderThread:600018 [sender.py:_start_run_threads():1188] run started: ob7chcle with start time 1733882654.256577 |
| 2024-12-10 21:04:14,564 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-10 21:04:14,564 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: check_version |
| 2024-12-10 21:04:14,612 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-10 21:04:14,617 DEBUG HandlerThread:600018 [system_info.py:__init__():26] System info init |
| 2024-12-10 21:04:14,617 DEBUG HandlerThread:600018 [system_info.py:__init__():41] System info init done |
| 2024-12-10 21:04:14,617 INFO HandlerThread:600018 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-10 21:04:14,617 INFO SystemMonitor:600018 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-10 21:04:14,617 INFO HandlerThread:600018 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-10 21:04:14,618 INFO SystemMonitor:600018 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-10 21:04:14,618 INFO SystemMonitor:600018 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-10 21:04:14,619 INFO SystemMonitor:600018 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-10 21:04:14,619 INFO SystemMonitor:600018 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-10 21:04:14,620 INFO SystemMonitor:600018 [interfaces.py:start():188] Started network monitoring |
| 2024-12-10 21:04:14,630 DEBUG HandlerThread:600018 [system_info.py:probe():152] Probing system |
| 2024-12-10 21:04:14,631 DEBUG HandlerThread:600018 [system_info.py:_probe_git():137] Probing git |
| 2024-12-10 21:04:14,636 DEBUG HandlerThread:600018 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-10 21:04:14,636 DEBUG HandlerThread:600018 [system_info.py:probe():200] Probing system done |
| 2024-12-10 21:04:14,636 DEBUG HandlerThread:600018 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-11T02:04:14.630135', 'startedAt': '2024-12-11T02:04:14.247012', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '4f7c0421eb84a7caf6bf5670f68777523362676f'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4124.996375000001, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.021, 'min': 800.0, 'max': 4700.0}, {'current': 4500.001, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.001, 'min': 800.0, 'max': 4700.0}, {'current': 4500.023, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 366.0494689941406}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71596908569336}} |
| 2024-12-10 21:04:14,636 INFO HandlerThread:600018 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-10 21:04:14,636 INFO HandlerThread:600018 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-10 21:04:14,636 DEBUG HandlerThread:600018 [system_info.py:_save_code():44] Saving code |
| 2024-12-10 21:04:14,638 DEBUG HandlerThread:600018 [system_info.py:_save_code():67] Saving code done |
| 2024-12-10 21:04:14,638 DEBUG HandlerThread:600018 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-10 21:04:14,703 DEBUG HandlerThread:600018 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-10 21:04:14,704 INFO HandlerThread:600018 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-10 21:04:14,707 DEBUG SenderThread:600018 [sender.py:send():379] send: files |
| 2024-12-10 21:04:14,707 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-10 21:04:14,708 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-10 21:04:14,708 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-10 21:04:14,959 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-10 21:04:14,960 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-10 21:04:14,960 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:04:14,960 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:04:14,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:14,963 INFO HandlerThread:600018 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/tb" |
| save: true |
| } |
| control { |
| relay_id: "5cdf21b441a841eeb077b1d066a521dc" |
| } |
| _info { |
| stream_id: "ob7chcle" |
| } |
|
|
| 2024-12-10 21:04:14,963 DEBUG HandlerThread:600018 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-10 21:04:14,988 INFO wandb-upload_0:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/vuiipzpx-wandb-metadata.json |
| 2024-12-10 21:04:15,027 DEBUG SenderThread:600018 [sender.py:send():379] send: telemetry |
| 2024-12-10 21:04:15,027 DEBUG SenderThread:600018 [sender.py:send():379] send: tbrecord |
| 2024-12-10 21:04:15,028 DEBUG SenderThread:600018 [sender.py:send():379] send: files |
| 2024-12-10 21:04:15,028 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file events.out.tfevents.1733882654.leonmkim-ROG-Strix-G15CS-G15CS.599894.0 with policy live |
| 2024-12-10 21:04:15,030 INFO wandb-upload_1:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/4jll3ix3-code/FISH/eval_robot.py |
| 2024-12-10 21:04:15,039 INFO wandb-upload_2:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/5pcafv76-diff.patch |
| 2024-12-10 21:04:15,220 INFO wandb-upload_0:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/o3h98mw2-events.out.tfevents.1733882654.leonmkim-ROG-Strix-G15CS-G15CS.599894.0 |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/requirements.txt |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/code/FISH/eval_robot.py |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-metadata.json |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/events.out.tfevents.1733882654.leonmkim-ROG-Strix-G15CS-G15CS.599894.0 |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/diff.patch |
| 2024-12-10 21:04:15,543 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/code/FISH |
| 2024-12-10 21:04:15,544 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/code |
| 2024-12-10 21:04:15,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:16,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:17,542 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:04:17,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:18,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:19,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:19,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:20,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:21,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:22,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:23,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:24,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:24,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:25,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:26,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:27,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:28,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:29,960 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:04:29,960 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:04:30,002 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:30,045 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:30,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:31,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:32,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:33,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:34,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:35,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:35,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:36,965 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:37,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:38,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:39,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:40,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:40,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:41,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:42,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:43,966 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:44,960 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:04:44,960 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:04:45,001 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:45,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:46,018 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:46,550 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/config.yaml |
| 2024-12-10 21:04:46,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:47,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:48,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:49,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:50,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:51,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:51,967 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:52,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:53,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:54,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:55,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:56,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:04:56,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:57,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:58,968 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:04:59,960 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:04:59,961 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:05:00,002 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:00,969 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:01,969 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:02,107 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:02,969 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:03,969 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:04,969 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:05,969 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:06,970 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:07,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:07,970 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:08,018 DEBUG SenderThread:600018 [sender.py:send():379] send: config |
| 2024-12-10 21:05:08,970 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:09,556 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:09,970 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:10,970 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:11,556 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:11,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:12,970 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:13,449 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:13,557 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:13,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:14,620 DEBUG SystemMonitor:600018 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-10 21:05:14,621 DEBUG SenderThread:600018 [sender.py:send():379] send: stats |
| 2024-12-10 21:05:14,960 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:05:14,961 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:05:15,002 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:15,558 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:15,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:16,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:17,558 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:17,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:18,600 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:18,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:19,559 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:19,559 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/config.yaml |
| 2024-12-10 21:05:19,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:20,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:21,559 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:21,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:22,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:23,560 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:23,971 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:24,064 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:24,972 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:25,560 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:25,972 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:26,972 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:27,561 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:27,972 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:28,973 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:29,124 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:29,562 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:29,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:05:29,961 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:05:29,973 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:30,973 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:31,562 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:31,973 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:32,975 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:33,563 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:33,975 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:34,275 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:34,975 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:35,563 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:35,976 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:36,976 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:37,564 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:37,976 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:38,976 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:39,457 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:39,564 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:39,976 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:40,976 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:41,565 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:41,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:42,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:43,566 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:43,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:44,621 DEBUG SenderThread:600018 [sender.py:send():379] send: stats |
| 2024-12-10 21:05:44,622 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:44,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:05:44,961 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:05:44,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:45,567 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:45,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:46,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:47,567 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:47,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:48,977 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:49,568 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:49,648 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:49,880 DEBUG SenderThread:600018 [sender.py:send():379] send: files |
| 2024-12-10 21:05:49,880 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_242331cb3a64f66021df.mp4 with policy now |
| 2024-12-10 21:05:49,880 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:05:49,881 DEBUG SenderThread:600018 [sender.py:send():379] send: history |
| 2024-12-10 21:05:49,881 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:05:49,882 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:05:49,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:50,165 INFO wandb-upload_1:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/dx8d7p0q-media/videos/eval/0_eval_0_242331cb3a64f66021df.mp4 |
| 2024-12-10 21:05:50,573 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos/eval/0_eval_0_242331cb3a64f66021df.mp4 |
| 2024-12-10 21:05:50,573 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-summary.json |
| 2024-12-10 21:05:50,573 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos/eval |
| 2024-12-10 21:05:50,573 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos |
| 2024-12-10 21:05:50,574 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media |
| 2024-12-10 21:05:50,912 INFO wandb-upload_0:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/64whhgw0-media/videos/eval/0_eval_0_242331cb3a64f66021df.mp4 |
| 2024-12-10 21:05:50,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:51,573 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:51,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:52,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:53,574 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:53,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:54,963 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:05:54,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:55,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:56,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:57,575 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:05:57,978 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:58,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:05:59,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:05:59,961 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:06:00,001 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:00,045 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:00,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:01,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:02,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:03,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:04,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:05,963 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:05,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:06,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:07,979 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:08,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:09,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:10,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:11,963 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:11,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:12,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:13,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:14,622 DEBUG SenderThread:600018 [sender.py:send():379] send: stats |
| 2024-12-10 21:06:14,961 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:06:14,961 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:06:15,001 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:15,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:16,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:17,057 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:17,980 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:18,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:19,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:20,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:21,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:22,963 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:22,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:23,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:24,981 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:25,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:26,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:27,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:27,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:28,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:29,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:06:29,962 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:06:30,005 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:30,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:31,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:32,982 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:33,045 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:33,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:34,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:35,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:36,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:37,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:38,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:38,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:39,983 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:40,984 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:41,984 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:42,984 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:43,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:43,985 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:44,622 DEBUG SenderThread:600018 [sender.py:send():379] send: stats |
| 2024-12-10 21:06:44,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:06:44,962 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:06:45,006 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:45,986 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:46,986 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:47,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:48,986 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:49,027 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:49,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:50,986 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:51,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:52,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:53,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:54,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:06:54,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:55,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:56,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:57,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:58,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:06:59,962 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-10 21:06:59,962 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-10 21:07:00,005 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:00,048 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:07:00,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:01,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:02,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:03,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:04,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:05,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:07:05,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:06,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:07,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:08,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:09,989 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:10,964 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:07:10,989 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:11,989 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:12,999 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:13,402 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-10 21:07:13,402 DEBUG SenderThread:600018 [sender.py:send():379] send: files |
| 2024-12-10 21:07:13,402 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file media/videos/eval/video_1_25445a41b5bbda0ed7e2.mp4 with policy now |
| 2024-12-10 21:07:13,403 DEBUG SenderThread:600018 [sender.py:send():379] send: history |
| 2024-12-10 21:07:13,403 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-10 21:07:13,403 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:07:13,404 DEBUG SenderThread:600018 [sender.py:send():379] send: exit |
| 2024-12-10 21:07:13,404 INFO SenderThread:600018 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-10 21:07:13,404 INFO SenderThread:600018 [sender.py:send_exit():588] handling runtime: 178 |
| 2024-12-10 21:07:13,404 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:07:13,404 INFO SenderThread:600018 [sender.py:send_exit():594] send defer |
| 2024-12-10 21:07:13,404 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:13,404 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-10 21:07:13,405 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:13,405 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-10 21:07:13,405 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-10 21:07:13,405 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:13,405 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-10 21:07:13,406 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:13,406 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-10 21:07:13,406 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-10 21:07:13,407 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:13,407 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-10 21:07:13,407 INFO HandlerThread:600018 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-10 21:07:13,407 DEBUG SystemMonitor:600018 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-10 21:07:13,407 DEBUG SystemMonitor:600018 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-10 21:07:13,407 INFO HandlerThread:600018 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-10 21:07:13,408 INFO HandlerThread:600018 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-10 21:07:13,456 INFO HandlerThread:600018 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-10 21:07:13,456 INFO HandlerThread:600018 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-10 21:07:13,456 INFO HandlerThread:600018 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-10 21:07:13,456 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:13,456 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-10 21:07:13,456 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-10 21:07:13,456 DEBUG SenderThread:600018 [sender.py:send():379] send: stats |
| 2024-12-10 21:07:13,456 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:13,457 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-10 21:07:13,457 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:13,457 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-10 21:07:13,457 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-10 21:07:13,457 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:13,457 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-10 21:07:13,599 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:07:13,600 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-summary.json |
| 2024-12-10 21:07:13,600 INFO Thread-12 :600018 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos/eval/video_1_25445a41b5bbda0ed7e2.mp4 |
| 2024-12-10 21:07:13,600 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos/eval |
| 2024-12-10 21:07:13,607 INFO wandb-upload_1:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/ycl94f2h-media/videos/eval/video_1_25445a41b5bbda0ed7e2.mp4 |
| 2024-12-10 21:07:13,892 INFO wandb-upload_0:600018 [upload_job.py:push():130] Uploaded file /tmp/tmpr0qlu59lwandb/53aux0ik-media/videos/eval/video_1_25445a41b5bbda0ed7e2.mp4 |
| 2024-12-10 21:07:15,600 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:07:19,985 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:07:19,985 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:19,985 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:07:19,986 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-10 21:07:19,986 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-10 21:07:19,986 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-10 21:07:19,986 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:07:19,986 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:19,986 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-10 21:07:19,987 DEBUG SenderThread:600018 [sender.py:send():379] send: summary |
| 2024-12-10 21:07:19,987 INFO SenderThread:600018 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-10 21:07:19,987 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:19,987 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-10 21:07:19,987 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-10 21:07:19,987 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:19,987 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-10 21:07:19,987 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:19,988 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-10 21:07:19,988 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-10 21:07:19,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-10 21:07:19,988 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:19,988 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-10 21:07:19,988 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:19,988 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-10 21:07:20,405 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:07:20,602 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-summary.json |
| 2024-12-10 21:07:21,263 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-10 21:07:21,263 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:07:21,263 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:21,263 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-10 21:07:21,263 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:21,263 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-10 21:07:21,263 INFO SenderThread:600018 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-10 21:07:21,266 INFO SenderThread:600018 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-10 21:07:21,280 INFO SenderThread:600018 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-10 21:07:21,294 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-10 21:07:21,295 DEBUG SenderThread:600018 [sender.py:send():379] send: artifact |
| 2024-12-10 21:07:21,295 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:21,296 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-10 21:07:21,405 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:07:21,602 INFO Thread-12 :600018 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:07:21,765 INFO wandb-upload_2:600018 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp0up_8ql4 |
| 2024-12-10 21:07:21,896 INFO wandb-upload_1:600018 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmphpmz9ew4 |
| 2024-12-10 21:07:21,956 INFO wandb-upload_0:600018 [upload_job.py:push():88] Uploaded file /tmp/tmpbv2i4zjr/wandb-job.json |
| 2024-12-10 21:07:22,564 INFO SenderThread:600018 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM2Njc3NTA2NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM2Njc2MTczOA==', 'versionIndex': 337}}} |
| 2024-12-10 21:07:22,564 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:22,564 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-10 21:07:22,564 INFO SenderThread:600018 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-10 21:07:22,603 INFO SenderThread:600018 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files |
| 2024-12-10 21:07:22,603 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/config.yaml config.yaml |
| 2024-12-10 21:07:22,603 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-summary.json wandb-summary.json |
| 2024-12-10 21:07:22,605 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/requirements.txt requirements.txt |
| 2024-12-10 21:07:22,605 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-10 21:07:22,606 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/events.out.tfevents.1733882654.leonmkim-ROG-Strix-G15CS-G15CS.599894.0 events.out.tfevents.1733882654.leonmkim-ROG-Strix-G15CS-G15CS.599894.0 |
| 2024-12-10 21:07:22,606 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log output.log |
| 2024-12-10 21:07:22,607 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/diff.patch diff.patch |
| 2024-12-10 21:07:22,608 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-10 21:07:22,610 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos/eval/0_eval_0_242331cb3a64f66021df.mp4 media/videos/eval/0_eval_0_242331cb3a64f66021df.mp4 |
| 2024-12-10 21:07:22,610 INFO SenderThread:600018 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/media/videos/eval/video_1_25445a41b5bbda0ed7e2.mp4 media/videos/eval/video_1_25445a41b5bbda0ed7e2.mp4 |
| 2024-12-10 21:07:22,610 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-10 21:07:22,610 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:07:22,610 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:22,610 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-10 21:07:22,610 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:22,610 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-10 21:07:22,610 INFO SenderThread:600018 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-10 21:07:22,797 INFO wandb-upload_0:600018 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/wandb-summary.json |
| 2024-12-10 21:07:22,818 INFO wandb-upload_1:600018 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/requirements.txt |
| 2024-12-10 21:07:22,829 INFO wandb-upload_2:600018 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/config.yaml |
| 2024-12-10 21:07:22,893 INFO wandb-upload_3:600018 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/files/output.log |
| 2024-12-10 21:07:23,093 INFO Thread-11 (_thread_body):600018 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-10 21:07:23,094 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:23,094 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-10 21:07:23,094 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:23,094 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-10 21:07:23,094 INFO SenderThread:600018 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-10 21:07:23,094 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-10 21:07:23,095 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:23,095 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-10 21:07:23,095 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:23,095 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-10 21:07:23,095 INFO SenderThread:600018 [file_stream.py:finish():601] file stream finish called |
| 2024-12-10 21:07:23,318 INFO SenderThread:600018 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-10 21:07:23,319 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-10 21:07:23,319 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:23,319 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-10 21:07:23,319 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:23,319 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-10 21:07:23,319 INFO SenderThread:600018 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-10 21:07:23,319 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-10 21:07:23,319 INFO HandlerThread:600018 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-10 21:07:23,319 DEBUG SenderThread:600018 [sender.py:send():379] send: final |
| 2024-12-10 21:07:23,320 DEBUG SenderThread:600018 [sender.py:send():379] send: footer |
| 2024-12-10 21:07:23,320 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: defer |
| 2024-12-10 21:07:23,320 INFO SenderThread:600018 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-10 21:07:23,320 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:07:23,320 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-10 21:07:23,320 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:07:23,321 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: server_info |
| 2024-12-10 21:07:23,322 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-10 21:07:23,322 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-10 21:07:23,323 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-10 21:07:23,323 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-10 21:07:23,446 DEBUG SenderThread:600018 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-10 21:07:23,446 INFO MainThread:600018 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-10 21:07:23,446 INFO MainThread:600018 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-10 21:07:23,446 INFO MainThread:600018 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-10 21:07:23,446 DEBUG HandlerThread:600018 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-10 21:07:23,446 INFO HandlerThread:600018 [handler.py:finish():882] shutting down handler |
| 2024-12-10 21:07:24,322 INFO WriterThread:600018 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265626/210407/wandb/run-20241210_210414-ob7chcle/run-ob7chcle.wandb |
| 2024-12-10 21:07:24,446 INFO SenderThread:600018 [sender.py:finish():1615] shutting down sender |
| 2024-12-10 21:07:24,446 INFO SenderThread:600018 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-10 21:07:24,446 INFO SenderThread:600018 [file_pusher.py:join():175] waiting for file pusher |
|
|
