| 2024-12-17 20:56:25,183 INFO StreamThr :70616 [internal.py:wandb_internal():85] W&B internal server running at pid: 70616, started at: 2024-12-17 20:56:25.182923 |
| 2024-12-17 20:56:25,184 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status |
| 2024-12-17 20:56:25,193 INFO WriterThread:70616 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/run-m7h1fbac.wandb |
| 2024-12-17 20:56:25,194 DEBUG SenderThread:70616 [sender.py:send():379] send: header |
| 2024-12-17 20:56:25,206 DEBUG SenderThread:70616 [sender.py:send():379] send: run |
| 2024-12-17 20:56:25,450 INFO SenderThread:70616 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files |
| 2024-12-17 20:56:25,450 INFO SenderThread:70616 [sender.py:_start_run_threads():1188] run started: m7h1fbac with start time 1734486985.183267 |
| 2024-12-17 20:56:25,468 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-17 20:56:25,468 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: check_version |
| 2024-12-17 20:56:25,518 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-17 20:56:25,523 DEBUG HandlerThread:70616 [system_info.py:__init__():26] System info init |
| 2024-12-17 20:56:25,523 DEBUG HandlerThread:70616 [system_info.py:__init__():41] System info init done |
| 2024-12-17 20:56:25,523 INFO HandlerThread:70616 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-17 20:56:25,523 INFO SystemMonitor:70616 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-17 20:56:25,523 INFO HandlerThread:70616 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-17 20:56:25,524 INFO SystemMonitor:70616 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-17 20:56:25,524 INFO SystemMonitor:70616 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-17 20:56:25,525 INFO SystemMonitor:70616 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-17 20:56:25,525 INFO SystemMonitor:70616 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-17 20:56:25,526 INFO SystemMonitor:70616 [interfaces.py:start():188] Started network monitoring |
| 2024-12-17 20:56:25,537 DEBUG HandlerThread:70616 [system_info.py:probe():152] Probing system |
| 2024-12-17 20:56:25,539 DEBUG HandlerThread:70616 [system_info.py:_probe_git():137] Probing git |
| 2024-12-17 20:56:25,543 DEBUG HandlerThread:70616 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-17 20:56:25,543 DEBUG HandlerThread:70616 [system_info.py:probe():200] Probing system done |
| 2024-12-17 20:56:25,543 DEBUG HandlerThread:70616 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T01:56:25.537675', 'startedAt': '2024-12-18T01:56:25.175931', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3570.471375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.958, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 367.0489959716797}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-17 20:56:25,543 INFO HandlerThread:70616 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-17 20:56:25,543 INFO HandlerThread:70616 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-17 20:56:25,544 DEBUG HandlerThread:70616 [system_info.py:_save_code():44] Saving code |
| 2024-12-17 20:56:25,545 DEBUG HandlerThread:70616 [system_info.py:_save_code():67] Saving code done |
| 2024-12-17 20:56:25,545 DEBUG HandlerThread:70616 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-17 20:56:25,610 DEBUG HandlerThread:70616 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-17 20:56:25,610 INFO HandlerThread:70616 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-17 20:56:25,613 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 20:56:25,613 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-17 20:56:25,613 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-17 20:56:25,614 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-17 20:56:25,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-17 20:56:25,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:56:25,843 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-17 20:56:25,843 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:56:25,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:25,846 INFO HandlerThread:70616 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/tb" |
| save: true |
| } |
| control { |
| relay_id: "19469a5384a5467b80ba8af284641d13" |
| } |
| _info { |
| stream_id: "m7h1fbac" |
| } |
|
|
| 2024-12-17 20:56:25,846 DEBUG HandlerThread:70616 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-17 20:56:25,873 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/fe6nodd0-wandb-metadata.json |
| 2024-12-17 20:56:25,876 INFO wandb-upload_2:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/yn4a2obv-diff.patch |
| 2024-12-17 20:56:25,907 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/e3g49mjo-code/FISH/eval_robot.py |
| 2024-12-17 20:56:25,963 DEBUG SenderThread:70616 [sender.py:send():379] send: telemetry |
| 2024-12-17 20:56:25,963 DEBUG SenderThread:70616 [sender.py:send():379] send: tbrecord |
| 2024-12-17 20:56:25,964 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 20:56:25,964 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 with policy live |
| 2024-12-17 20:56:26,162 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/wkffbmz1-events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:56:26,451 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/code/FISH/eval_robot.py |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-metadata.json |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/requirements.txt |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/diff.patch |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/code |
| 2024-12-17 20:56:26,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/code/FISH |
| 2024-12-17 20:56:26,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:27,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:28,451 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:28,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:29,700 DEBUG SenderThread:70616 [sender.py:send():379] send: config |
| 2024-12-17 20:56:29,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:30,706 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:56:30,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:31,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:32,452 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:32,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:33,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:34,453 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:34,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:35,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:56:35,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:36,453 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:36,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:37,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:38,454 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:38,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:39,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:40,455 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:40,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:56:40,843 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:56:40,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:40,910 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:56:41,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:42,455 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:42,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:43,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:44,456 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:44,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:45,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:46,456 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:46,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:56:46,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:47,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:48,457 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:48,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:49,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:50,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:51,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:51,879 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:56:52,458 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:52,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:53,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:54,459 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:56:54,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:55,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:56:55,843 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:56:55,889 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:56,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:56,957 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:56:57,460 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/config.yaml |
| 2024-12-17 20:56:57,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:58,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:56:59,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:00,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:01,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:02,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:02,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:03,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:04,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:05,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:06,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:07,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:07,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:08,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:09,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:10,452 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 20:57:10,452 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_1ac69680a9ec173c38e2.mp4 with policy now |
| 2024-12-17 20:57:10,452 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 20:57:10,453 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:57:10,453 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:57:10,454 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:57:10,463 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:10,463 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/0_eval_0_1ac69680a9ec173c38e2.mp4 |
| 2024-12-17 20:57:10,464 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:57:10,464 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval |
| 2024-12-17 20:57:10,464 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos |
| 2024-12-17 20:57:10,464 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media |
| 2024-12-17 20:57:10,710 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/2m43vjbj-media/videos/eval/0_eval_0_1ac69680a9ec173c38e2.mp4 |
| 2024-12-17 20:57:10,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:57:10,844 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:57:10,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:10,971 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/gxg5gj2f-media/videos/eval/0_eval_0_1ac69680a9ec173c38e2.mp4 |
| 2024-12-17 20:57:11,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:12,464 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:12,850 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:12,984 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:13,851 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:14,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:15,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:16,465 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:16,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:17,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:18,466 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:18,821 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:18,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:19,466 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:57:19,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:20,467 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:20,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:21,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:22,467 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:22,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:23,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:24,348 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:24,468 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:24,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:25,526 DEBUG SystemMonitor:70616 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-17 20:57:25,526 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 20:57:25,843 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:57:25,844 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:57:25,885 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:26,854 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:27,854 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:28,854 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:29,854 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:29,966 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:30,854 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:31,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:32,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:33,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:34,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:35,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:35,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:36,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:37,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:38,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:39,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:40,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:57:40,844 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:57:40,889 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:40,932 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:41,855 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:42,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:43,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:44,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:45,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:46,847 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:46,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:47,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:48,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:49,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:50,856 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:51,053 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 20:57:51,054 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 20:57:51,054 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_a59176de7339dfa56b5f.mp4 with policy now |
| 2024-12-17 20:57:51,054 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:57:51,054 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:57:51,055 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:57:51,308 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/gwsratzy-media/videos/eval/1_eval_1_a59176de7339dfa56b5f.mp4 |
| 2024-12-17 20:57:51,476 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:57:51,477 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/1_eval_1_a59176de7339dfa56b5f.mp4 |
| 2024-12-17 20:57:51,477 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval |
| 2024-12-17 20:57:51,732 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/00fbg6z1-media/videos/eval/1_eval_1_a59176de7339dfa56b5f.mp4 |
| 2024-12-17 20:57:51,863 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:52,056 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:52,477 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:52,863 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:53,862 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:54,863 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:55,527 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 20:57:55,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:57:55,848 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:57:55,889 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:56,478 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:56,869 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:57,394 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:57:57,866 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:58,479 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:57:58,871 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:57:59,871 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:00,479 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:58:00,723 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/b5x3uf8q-events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:58:00,867 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:01,239 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:58:01,239 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:58:01,240 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:58:01,480 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:58:01,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:02,480 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:02,873 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:03,241 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:03,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:04,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:05,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:06,481 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:06,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:07,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:08,482 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:08,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:09,049 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:09,868 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:10,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:58:10,849 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:58:10,869 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:11,869 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:12,484 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:12,869 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:13,870 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:14,485 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:14,870 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:15,049 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:15,870 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:16,485 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:16,871 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:17,871 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:18,870 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:19,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:20,487 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:20,849 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:20,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:21,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:22,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:23,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:24,488 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:24,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:25,527 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 20:58:25,844 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:58:25,845 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:58:25,885 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:25,903 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:26,872 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:27,873 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:28,873 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:29,873 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:30,873 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:31,398 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:31,654 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 20:58:31,654 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_eb964b6ad7f53c112f71.mp4 with policy now |
| 2024-12-17 20:58:31,657 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 20:58:31,657 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:58:31,657 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:58:31,658 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:58:31,874 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:31,907 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/bz7uu82v-media/videos/eval/2_eval_2_eb964b6ad7f53c112f71.mp4 |
| 2024-12-17 20:58:32,491 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:32,491 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:58:32,491 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/2_eval_2_eb964b6ad7f53c112f71.mp4 |
| 2024-12-17 20:58:32,491 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval |
| 2024-12-17 20:58:32,796 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/u5o9rvmq-media/videos/eval/2_eval_2_eb964b6ad7f53c112f71.mp4 |
| 2024-12-17 20:58:32,876 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:33,881 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:34,491 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:34,875 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:35,881 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:36,492 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:36,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:36,882 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:37,889 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:38,884 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:39,884 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:40,493 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:40,846 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:58:40,847 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:58:40,889 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:41,886 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:41,984 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:42,888 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:43,887 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:44,887 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:45,495 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:58:45,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:46,061 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:58:46,063 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:58:46,064 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:58:46,495 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:46,495 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:58:46,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:47,225 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:47,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:48,496 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:48,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:49,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:50,497 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:58:50,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:51,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:52,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:52,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:53,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:54,890 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:55,528 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 20:58:55,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:58:55,845 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:58:55,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:56,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:57,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:57,920 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:58:58,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:58:59,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:00,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:01,891 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:02,892 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:03,852 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:03,892 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:04,892 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:05,892 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:06,892 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:07,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:08,853 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:08,894 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:09,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:10,845 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:59:10,845 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:59:10,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:11,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:12,504 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:12,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:13,893 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:13,931 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:14,894 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:15,894 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:16,505 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:16,871 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 20:59:16,871 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_d1fd18c7f97afc0352cb.mp4 with policy now |
| 2024-12-17 20:59:16,879 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 20:59:16,880 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:59:16,880 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:59:16,880 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:59:16,915 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:17,120 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/zwvwl4qx-media/videos/eval/3_eval_3_d1fd18c7f97afc0352cb.mp4 |
| 2024-12-17 20:59:17,506 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:59:17,506 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/3_eval_3_d1fd18c7f97afc0352cb.mp4 |
| 2024-12-17 20:59:17,506 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval |
| 2024-12-17 20:59:17,769 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/5yfl7pkc-media/videos/eval/3_eval_3_d1fd18c7f97afc0352cb.mp4 |
| 2024-12-17 20:59:17,916 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:18,506 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:18,904 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:19,892 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:19,925 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:20,917 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:21,912 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:22,507 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:22,912 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:23,914 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:24,508 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:24,919 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:25,373 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:25,528 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 20:59:25,850 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:59:25,850 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:59:25,917 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:26,916 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:27,920 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:28,923 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:29,924 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:30,870 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:30,919 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:31,920 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:32,927 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:33,928 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:34,924 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:35,923 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:36,867 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:36,921 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:37,922 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:38,929 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:39,931 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:40,863 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:59:40,863 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:59:40,926 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:41,929 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:41,989 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:42,931 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:43,941 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:44,932 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:45,942 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:46,515 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:59:46,775 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/4e06pnb5-events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 20:59:46,935 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:47,138 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 20:59:47,138 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 20:59:47,139 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:47,140 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 20:59:47,515 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 20:59:47,932 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:48,516 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:48,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:49,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:50,932 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:51,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:52,517 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 20:59:52,865 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:52,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:53,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:54,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:55,529 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 20:59:55,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 20:59:55,848 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 20:59:55,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:56,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:57,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:57,987 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 20:59:58,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 20:59:59,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:00,933 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:01,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:02,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:03,865 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:03,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:04,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:05,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:06,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:07,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:08,522 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:08,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:08,979 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:09,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:10,523 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:10,848 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:00:10,848 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:00:10,934 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:11,935 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:12,935 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:13,935 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:14,525 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:14,779 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:14,935 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:15,935 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:16,525 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:16,937 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:17,953 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:18,526 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:18,693 DEBUG SenderThread:70616 [sender.py:send():379] send: files |
| 2024-12-17 21:00:18,693 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_cc34baaf71385e46f645.mp4 with policy now |
| 2024-12-17 21:00:18,696 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:00:18,696 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 21:00:18,696 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:00:18,696 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:00:18,952 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/l2dpc4od-media/videos/eval/4_eval_4_cc34baaf71385e46f645.mp4 |
| 2024-12-17 21:00:18,955 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:19,526 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 21:00:19,527 INFO Thread-12 :70616 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/4_eval_4_cc34baaf71385e46f645.mp4 |
| 2024-12-17 21:00:19,527 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval |
| 2024-12-17 21:00:19,804 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/c179dvnw-media/videos/eval/4_eval_4_cc34baaf71385e46f645.mp4 |
| 2024-12-17 21:00:19,952 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:20,527 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:20,716 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:20,951 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:21,527 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:21,956 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:22,970 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:23,952 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:24,528 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:24,955 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:25,529 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 21:00:25,864 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:00:25,864 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:00:25,955 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:25,962 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:26,529 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:26,961 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:27,613 DEBUG SenderThread:70616 [sender.py:send():379] send: exit |
| 2024-12-17 21:00:27,613 INFO SenderThread:70616 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-17 21:00:27,613 INFO SenderThread:70616 [sender.py:send_exit():588] handling runtime: 242 |
| 2024-12-17 21:00:27,614 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:00:27,614 INFO SenderThread:70616 [sender.py:send_exit():594] send defer |
| 2024-12-17 21:00:27,614 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:27,614 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-17 21:00:27,614 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:27,614 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-17 21:00:27,614 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-17 21:00:27,614 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:27,614 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-17 21:00:27,614 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:27,614 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-17 21:00:27,614 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-17 21:00:27,615 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:27,615 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-17 21:00:27,615 INFO HandlerThread:70616 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-17 21:00:27,615 DEBUG SystemMonitor:70616 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-17 21:00:27,615 DEBUG SystemMonitor:70616 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-17 21:00:27,615 INFO HandlerThread:70616 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-17 21:00:27,616 INFO HandlerThread:70616 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-17 21:00:27,679 INFO HandlerThread:70616 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-17 21:00:27,679 INFO HandlerThread:70616 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-17 21:00:27,679 INFO HandlerThread:70616 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-17 21:00:27,680 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:27,680 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-17 21:00:27,680 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-17 21:00:27,680 DEBUG SenderThread:70616 [sender.py:send():379] send: stats |
| 2024-12-17 21:00:27,680 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:27,680 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-17 21:00:27,680 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:27,680 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-17 21:00:27,680 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-17 21:00:27,680 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:27,680 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-17 21:00:28,530 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:28,530 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 21:00:28,530 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 21:00:33,682 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:00:33,683 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:33,683 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-17 21:00:33,683 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:33,683 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-17 21:00:33,683 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-17 21:00:33,683 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 21:00:33,683 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:33,684 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:00:33,684 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-17 21:00:33,684 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:00:33,684 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:00:33,684 DEBUG SenderThread:70616 [sender.py:send():379] send: history |
| 2024-12-17 21:00:33,685 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:00:33,688 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:00:33,688 DEBUG SenderThread:70616 [sender.py:send():379] send: summary |
| 2024-12-17 21:00:33,689 INFO SenderThread:70616 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:00:33,689 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:33,689 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-17 21:00:33,689 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-17 21:00:33,689 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:33,689 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-17 21:00:33,689 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:33,689 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-17 21:00:33,689 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-17 21:00:33,690 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:00:33,690 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:33,690 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-17 21:00:33,690 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:33,690 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-17 21:00:34,532 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 21:00:34,602 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-17 21:00:34,602 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:34,602 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-17 21:00:34,602 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:34,603 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-17 21:00:34,603 INFO SenderThread:70616 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-17 21:00:34,605 INFO SenderThread:70616 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-17 21:00:34,614 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:00:34,619 INFO SenderThread:70616 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-17 21:00:34,632 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-17 21:00:34,632 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:00:34,632 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:34,632 DEBUG SenderThread:70616 [sender.py:send():379] send: artifact |
| 2024-12-17 21:00:34,632 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-17 21:00:35,321 INFO wandb-upload_2:70616 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpu4l34lz7 |
| 2024-12-17 21:00:35,321 INFO wandb-upload_0:70616 [upload_job.py:push():85] Skipped uploading /tmp/tmpthz7ujbr/wandb-job.json |
| 2024-12-17 21:00:35,452 INFO wandb-upload_1:70616 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpouixkgus |
| 2024-12-17 21:00:35,533 INFO Thread-12 :70616 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:35,615 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:00:36,051 INFO SenderThread:70616 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzA4NzA1NA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4MzA1OTA4Nw==', 'versionIndex': 374}}} |
| 2024-12-17 21:00:36,051 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:36,051 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-17 21:00:36,051 INFO SenderThread:70616 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-17 21:00:36,534 INFO SenderThread:70616 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files |
| 2024-12-17 21:00:36,534 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/config.yaml config.yaml |
| 2024-12-17 21:00:36,534 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json wandb-summary.json |
| 2024-12-17 21:00:36,534 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/requirements.txt requirements.txt |
| 2024-12-17 21:00:36,534 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-17 21:00:36,537 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 21:00:36,537 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log output.log |
| 2024-12-17 21:00:36,537 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/diff.patch diff.patch |
| 2024-12-17 21:00:36,539 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-17 21:00:36,539 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/0_eval_0_1ac69680a9ec173c38e2.mp4 media/videos/eval/0_eval_0_1ac69680a9ec173c38e2.mp4 |
| 2024-12-17 21:00:36,539 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/4_eval_4_cc34baaf71385e46f645.mp4 media/videos/eval/4_eval_4_cc34baaf71385e46f645.mp4 |
| 2024-12-17 21:00:36,540 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/2_eval_2_eb964b6ad7f53c112f71.mp4 media/videos/eval/2_eval_2_eb964b6ad7f53c112f71.mp4 |
| 2024-12-17 21:00:36,541 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/3_eval_3_d1fd18c7f97afc0352cb.mp4 media/videos/eval/3_eval_3_d1fd18c7f97afc0352cb.mp4 |
| 2024-12-17 21:00:36,541 INFO SenderThread:70616 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/media/videos/eval/1_eval_1_a59176de7339dfa56b5f.mp4 media/videos/eval/1_eval_1_a59176de7339dfa56b5f.mp4 |
| 2024-12-17 21:00:36,542 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-17 21:00:36,542 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:00:36,543 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:36,543 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-17 21:00:36,543 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:36,543 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-17 21:00:36,543 INFO SenderThread:70616 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-17 21:00:36,616 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:00:36,616 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:00:36,718 INFO wandb-upload_0:70616 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/wandb-summary.json |
| 2024-12-17 21:00:36,731 INFO wandb-upload_2:70616 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/config.yaml |
| 2024-12-17 21:00:37,118 INFO wandb-upload_1:70616 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/requirements.txt |
| 2024-12-17 21:00:37,154 INFO wandb-upload_3:70616 [upload_job.py:push():130] Uploaded file /tmp/tmpd0e87ne3wandb/igganeue-events.out.tfevents.1734486985.leonmkim-ROG-Strix-G15CS-G15CS.70432.0 |
| 2024-12-17 21:00:37,157 INFO wandb-upload_4:70616 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/files/output.log |
| 2024-12-17 21:00:37,357 INFO Thread-11 (_thread_body):70616 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-17 21:00:37,357 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:37,357 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-17 21:00:37,358 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:37,358 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-17 21:00:37,358 INFO SenderThread:70616 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-17 21:00:37,359 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-17 21:00:37,359 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:37,359 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-17 21:00:37,359 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:37,359 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-17 21:00:37,359 INFO SenderThread:70616 [file_stream.py:finish():601] file stream finish called |
| 2024-12-17 21:00:37,414 INFO SenderThread:70616 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-17 21:00:37,414 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-17 21:00:37,414 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:37,414 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-17 21:00:37,414 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:37,414 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-17 21:00:37,414 INFO SenderThread:70616 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-17 21:00:37,415 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:00:37,415 DEBUG SenderThread:70616 [sender.py:send():379] send: final |
| 2024-12-17 21:00:37,415 INFO HandlerThread:70616 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-17 21:00:37,415 DEBUG SenderThread:70616 [sender.py:send():379] send: footer |
| 2024-12-17 21:00:37,415 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:00:37,415 INFO SenderThread:70616 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-17 21:00:37,415 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:00:37,415 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:00:37,416 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-17 21:00:37,416 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: server_info |
| 2024-12-17 21:00:37,417 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:00:37,417 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-17 21:00:37,418 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-17 21:00:37,418 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:00:37,497 DEBUG SenderThread:70616 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:00:37,497 INFO MainThread:70616 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-17 21:00:37,497 INFO MainThread:70616 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-17 21:00:37,497 INFO MainThread:70616 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-17 21:00:37,498 DEBUG HandlerThread:70616 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-17 21:00:37,498 INFO HandlerThread:70616 [handler.py:finish():882] shutting down handler |
| 2024-12-17 21:00:38,418 INFO WriterThread:70616 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/506_0/205619/wandb/run-20241217_205625-m7h1fbac/run-m7h1fbac.wandb |
| 2024-12-17 21:00:38,497 INFO SenderThread:70616 [sender.py:finish():1615] shutting down sender |
| 2024-12-17 21:00:38,497 INFO SenderThread:70616 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-17 21:00:38,497 INFO SenderThread:70616 [file_pusher.py:join():175] waiting for file pusher |
|
|
