| 2024-12-31 23:22:51,413 INFO StreamThr :1764142 [internal.py:wandb_internal():85] W&B internal server running at pid: 1764142, started at: 2024-12-31 23:22:51.412894 |
| 2024-12-31 23:22:51,415 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status |
| 2024-12-31 23:22:51,427 INFO WriterThread:1764142 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/run-ix23zko1.wandb |
| 2024-12-31 23:22:51,428 DEBUG SenderThread:1764142 [sender.py:send():379] send: header |
| 2024-12-31 23:22:51,438 DEBUG SenderThread:1764142 [sender.py:send():379] send: run |
| 2024-12-31 23:22:51,672 INFO SenderThread:1764142 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files |
| 2024-12-31 23:22:51,673 INFO SenderThread:1764142 [sender.py:_start_run_threads():1188] run started: ix23zko1 with start time 1735705371.413732 |
| 2024-12-31 23:22:51,706 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-31 23:22:51,707 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: check_version |
| 2024-12-31 23:22:51,750 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-31 23:22:51,755 DEBUG HandlerThread:1764142 [system_info.py:__init__():26] System info init |
| 2024-12-31 23:22:51,755 DEBUG HandlerThread:1764142 [system_info.py:__init__():41] System info init done |
| 2024-12-31 23:22:51,755 INFO HandlerThread:1764142 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-31 23:22:51,755 INFO SystemMonitor:1764142 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-31 23:22:51,755 INFO HandlerThread:1764142 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-31 23:22:51,756 INFO SystemMonitor:1764142 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-31 23:22:51,756 INFO SystemMonitor:1764142 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-31 23:22:51,757 INFO SystemMonitor:1764142 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-31 23:22:51,757 INFO SystemMonitor:1764142 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-31 23:22:51,758 INFO SystemMonitor:1764142 [interfaces.py:start():188] Started network monitoring |
| 2024-12-31 23:22:51,769 DEBUG HandlerThread:1764142 [system_info.py:probe():152] Probing system |
| 2024-12-31 23:22:51,771 DEBUG HandlerThread:1764142 [system_info.py:_probe_git():137] Probing git |
| 2024-12-31 23:22:51,777 DEBUG HandlerThread:1764142 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-31 23:22:51,777 DEBUG HandlerThread:1764142 [system_info.py:probe():200] Probing system done |
| 2024-12-31 23:22:51,777 DEBUG HandlerThread:1764142 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-01T04:22:51.769978', 'startedAt': '2025-01-01T04:22:51.405701', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'efd33a493dbad5ba8b9c770955ac9f0942bd971a'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 4124.998375, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.019, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.002, 'min': 800.0, 'max': 4700.0}, {'current': 4500.015, 'min': 800.0, 'max': 4700.0}, {'current': 4500.002, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 579.1383628845215}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-31 23:22:51,777 INFO HandlerThread:1764142 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-31 23:22:51,777 INFO HandlerThread:1764142 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-31 23:22:51,777 DEBUG HandlerThread:1764142 [system_info.py:_save_code():44] Saving code |
| 2024-12-31 23:22:51,779 DEBUG HandlerThread:1764142 [system_info.py:_save_code():67] Saving code done |
| 2024-12-31 23:22:51,779 DEBUG HandlerThread:1764142 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-31 23:22:51,841 DEBUG HandlerThread:1764142 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-31 23:22:51,842 INFO HandlerThread:1764142 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-31 23:22:51,845 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:22:51,845 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-31 23:22:51,845 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-31 23:22:51,845 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-31 23:22:52,098 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-31 23:22:52,098 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-31 23:22:52,098 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:22:52,099 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:22:52,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:52,102 INFO HandlerThread:1764142 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/tb" |
| save: true |
| } |
| control { |
| relay_id: "383dac9a791f486ca65aa2a860b70512" |
| } |
| _info { |
| stream_id: "ix23zko1" |
| } |
|
|
| 2024-12-31 23:22:52,102 DEBUG HandlerThread:1764142 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-31 23:22:52,112 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/a3p5h8pv-wandb-metadata.json |
| 2024-12-31 23:22:52,119 INFO wandb-upload_2:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/7w2ikdem-diff.patch |
| 2024-12-31 23:22:52,173 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/y1cyo7vq-code/FISH/eval_robot.py |
| 2024-12-31 23:22:52,250 DEBUG SenderThread:1764142 [sender.py:send():379] send: telemetry |
| 2024-12-31 23:22:52,250 DEBUG SenderThread:1764142 [sender.py:send():379] send: tbrecord |
| 2024-12-31 23:22:52,251 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:22:52,251 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 with policy live |
| 2024-12-31 23:22:52,473 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/yhwx3t7z-events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:22:52,674 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/requirements.txt |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/code/FISH/eval_robot.py |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/diff.patch |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-metadata.json |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/code/FISH |
| 2024-12-31 23:22:52,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/code |
| 2024-12-31 23:22:53,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:54,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:54,674 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:22:55,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:56,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:57,062 DEBUG SenderThread:1764142 [sender.py:send():379] send: config |
| 2024-12-31 23:22:57,062 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:22:57,109 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:58,117 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:22:58,675 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:22:59,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:00,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:00,676 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:01,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:02,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:02,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:02,676 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:03,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:04,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:04,677 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:05,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:06,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:06,678 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:07,098 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:23:07,098 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:23:07,141 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:07,254 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:08,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:09,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:10,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:11,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:12,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:13,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:13,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:14,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:15,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:16,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:17,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:18,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:18,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:19,103 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:20,103 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:20,682 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:21,103 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:22,098 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:23:22,098 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:23:22,141 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:22,683 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:23,103 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:23,159 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:23,684 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/config.yaml |
| 2024-12-31 23:23:24,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:24,684 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:25,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:26,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:27,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:28,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:29,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:29,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:30,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:31,112 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:32,112 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:33,112 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:34,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:34,112 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:35,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:36,112 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:37,098 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:23:37,099 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:23:37,113 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:38,113 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:38,693 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:39,113 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:39,189 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:40,114 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:41,113 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:42,113 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:43,114 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:44,114 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:44,193 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:44,695 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:45,114 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:46,114 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:46,695 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:47,116 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:48,116 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:49,116 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:50,116 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:50,173 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:50,697 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:51,116 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:51,542 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:23:51,542 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_eeec50fdf1118de79305.mp4 with policy now |
| 2024-12-31 23:23:51,543 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:23:51,544 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:23:51,544 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:23:51,544 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:23:51,697 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:23:51,698 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/0_eval_0_eeec50fdf1118de79305.mp4 |
| 2024-12-31 23:23:51,698 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media |
| 2024-12-31 23:23:51,698 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval |
| 2024-12-31 23:23:51,698 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos |
| 2024-12-31 23:23:51,758 DEBUG SystemMonitor:1764142 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-31 23:23:51,759 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:23:51,807 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/cb3bkrvo-media/videos/eval/0_eval_0_eeec50fdf1118de79305.mp4 |
| 2024-12-31 23:23:52,067 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/muwwa1t9-media/videos/eval/0_eval_0_eeec50fdf1118de79305.mp4 |
| 2024-12-31 23:23:52,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:23:52,099 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:23:52,116 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:52,698 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:53,117 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:54,117 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:55,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:55,200 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:23:56,118 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:56,699 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:23:57,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:58,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:23:59,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:00,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:00,700 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:01,010 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:01,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:02,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:03,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:04,120 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:05,120 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:06,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:06,120 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:07,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:24:07,099 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:24:07,141 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:08,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:09,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:10,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:11,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:11,197 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:12,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:13,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:14,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:14,705 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:24:15,121 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:15,455 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/nzlocrr2-events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:24:16,122 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:16,537 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:16,706 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:17,122 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:18,122 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:19,122 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:20,122 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:20,707 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:21,123 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:21,766 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:24:21,766 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:22,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:24:22,099 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:24:22,145 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:23,123 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:24,123 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:25,123 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:26,123 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:27,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:27,123 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:28,124 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:29,124 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:30,124 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:31,124 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:32,125 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:33,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:33,125 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:34,125 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:35,126 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:36,126 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:36,712 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:37,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:24:37,099 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:24:37,145 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:38,126 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:38,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:39,126 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:40,129 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:41,128 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:42,128 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:43,129 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:44,102 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:44,129 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:45,129 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:46,129 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:46,716 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:46,824 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:24:46,825 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_00cd606e45f670b2df9c.mp4 with policy now |
| 2024-12-31 23:24:46,828 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:24:46,829 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:24:46,829 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:24:46,830 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:24:47,095 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/gauz3vyo-media/videos/eval/1_eval_1_00cd606e45f670b2df9c.mp4 |
| 2024-12-31 23:24:47,130 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:47,716 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:24:47,716 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/1_eval_1_00cd606e45f670b2df9c.mp4 |
| 2024-12-31 23:24:47,716 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval |
| 2024-12-31 23:24:47,999 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/gnwggf25-media/videos/eval/1_eval_1_00cd606e45f670b2df9c.mp4 |
| 2024-12-31 23:24:48,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:48,716 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:49,103 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:49,138 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:50,131 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:51,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:51,760 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:24:52,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:24:52,102 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:24:52,145 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:52,718 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:53,138 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:54,133 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:54,385 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:24:54,718 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:24:55,133 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:56,133 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:57,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:58,133 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:24:59,134 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:00,104 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:00,134 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:01,134 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:01,530 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:25:01,530 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:25:01,532 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:25:01,721 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:25:01,721 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:25:02,134 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:02,721 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:03,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:04,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:04,722 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:05,132 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:05,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:06,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:06,722 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:07,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:25:07,100 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:25:07,145 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:08,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:09,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:10,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:10,237 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:11,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:12,135 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:13,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:14,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:14,725 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:15,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:16,065 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:16,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:17,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:18,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:19,136 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:20,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:21,104 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:21,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:21,760 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:25:22,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:25:22,100 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:25:22,141 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:23,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:24,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:25,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:26,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:26,254 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:27,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:28,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:29,137 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:30,138 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:31,138 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:32,099 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:32,146 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:33,149 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:33,195 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:25:33,195 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_3ca0a8506093efa659f9.mp4 with policy now |
| 2024-12-31 23:25:33,198 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:25:33,199 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:25:33,199 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:25:33,199 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:25:33,452 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/gimsickx-media/videos/eval/2_eval_2_3ca0a8506093efa659f9.mp4 |
| 2024-12-31 23:25:33,731 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:25:33,732 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/2_eval_2_3ca0a8506093efa659f9.mp4 |
| 2024-12-31 23:25:33,732 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval |
| 2024-12-31 23:25:34,009 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/sxvw80v3-media/videos/eval/2_eval_2_3ca0a8506093efa659f9.mp4 |
| 2024-12-31 23:25:34,147 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:34,732 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:35,149 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:36,149 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:37,103 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:25:37,103 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:25:37,151 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:37,254 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:38,149 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:38,733 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:39,153 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:40,149 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:40,734 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:41,151 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:42,153 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:43,110 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:43,155 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:44,154 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:45,155 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:46,157 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:47,157 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:48,111 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:48,156 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:49,159 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:50,159 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:51,161 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:51,738 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:25:51,761 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:25:52,000 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/xbzcagni-events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:25:52,104 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:25:52,105 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:25:52,163 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:52,378 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:25:52,378 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:25:52,379 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:25:52,738 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:25:52,738 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:53,161 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:53,497 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:54,159 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:54,739 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:55,167 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:56,159 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:56,740 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:25:57,159 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:58,162 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:25:59,107 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:25:59,162 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:00,163 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:01,176 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:02,173 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:03,173 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:04,107 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:04,173 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:05,175 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:06,175 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:07,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:26:07,101 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:26:07,183 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:08,183 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:09,183 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:09,213 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:10,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:11,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:12,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:13,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:14,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:15,109 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:15,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:16,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:17,184 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:18,185 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:19,185 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:20,185 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:21,108 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:21,185 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:21,761 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:26:22,100 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:26:22,100 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:26:22,185 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:23,200 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:23,746 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:26:23,746 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_4b8025834b9f25779448.mp4 with policy now |
| 2024-12-31 23:26:23,749 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:26:23,749 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:26:23,749 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:26:23,750 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:26:23,750 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:26:23,750 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/3_eval_3_4b8025834b9f25779448.mp4 |
| 2024-12-31 23:26:23,750 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval |
| 2024-12-31 23:26:24,035 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/cr606cpd-media/videos/eval/3_eval_3_4b8025834b9f25779448.mp4 |
| 2024-12-31 23:26:24,194 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:24,332 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/2nit8asm-media/videos/eval/3_eval_3_4b8025834b9f25779448.mp4 |
| 2024-12-31 23:26:24,750 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:26:25,194 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:26,195 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:26,779 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:27,194 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:28,197 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:28,752 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:26:29,199 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:30,204 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:31,203 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:32,119 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:32,207 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:32,753 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:26:33,203 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:34,203 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:35,203 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:36,208 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:37,105 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:26:37,105 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:26:37,204 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:37,214 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:38,213 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:39,207 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:40,211 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:41,214 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:42,220 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:43,118 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:43,220 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:44,219 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:45,224 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:46,218 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:47,225 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:48,125 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:48,225 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:49,220 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:50,219 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:51,223 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:51,762 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:26:52,106 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:26:52,106 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:26:52,224 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:53,205 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:53,224 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:54,230 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:55,225 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:56,227 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:57,228 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:58,226 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:26:59,124 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:26:59,227 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:00,228 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:01,229 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:02,231 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:03,236 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:03,460 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:27:03,461 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:03,462 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:03,764 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:27:03,764 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:27:04,020 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/kssd9jo2-events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:27:04,236 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:04,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:04,764 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:05,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:06,237 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:06,765 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:07,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:27:07,102 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:27:07,238 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:08,239 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:08,766 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:09,234 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:09,903 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:10,234 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:10,766 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:11,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:12,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:13,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:14,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:15,120 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:15,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:16,235 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:17,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:18,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:19,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:20,120 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:20,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:21,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:21,762 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:27:22,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:27:22,101 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:27:22,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:23,245 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:24,246 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:25,262 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:25,316 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:25,362 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:27:25,362 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_e48861ffe574038dd2b6.mp4 with policy now |
| 2024-12-31 23:27:25,367 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:27:25,368 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:27:25,368 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:25,368 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:25,635 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/xfrfhtq1-media/videos/eval/4_eval_4_e48861ffe574038dd2b6.mp4 |
| 2024-12-31 23:27:25,771 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:27:25,772 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/4_eval_4_e48861ffe574038dd2b6.mp4 |
| 2024-12-31 23:27:25,772 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval |
| 2024-12-31 23:27:26,031 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/jza0ki16-media/videos/eval/4_eval_4_e48861ffe574038dd2b6.mp4 |
| 2024-12-31 23:27:26,252 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:26,772 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:27,260 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:28,258 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:29,261 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:30,265 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:30,773 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:31,045 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:31,269 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:32,266 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:32,774 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:33,264 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:34,267 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:34,775 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:35,274 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:36,122 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:36,266 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:37,106 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:27:37,106 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:27:37,268 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:38,265 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:39,269 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:40,269 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:41,191 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:41,279 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:42,276 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:43,268 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:44,276 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:45,280 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:46,283 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:47,131 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:47,285 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:48,284 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:49,288 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:50,283 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:50,447 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 23:27:50,447 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 23:27:50,643 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:27:50,780 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:27:50,781 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:50,781 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:27:50,781 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media |
| 2024-12-31 23:27:50,781 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table |
| 2024-12-31 23:27:50,781 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table/eval |
| 2024-12-31 23:27:50,894 INFO wandb-upload_0:1764142 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpk7287tw1 |
| 2024-12-31 23:27:50,979 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/len2tr5z-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:27:51,007 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: log_artifact |
| 2024-12-31 23:27:51,016 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-31 23:27:51,280 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:51,310 INFO SenderThread:1764142 [sender.py:send_request_log_artifact():1525] logged artifact run-ix23zko1-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDE5NzgzOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE3ODA3Nw==', 'latestArtifact': None}} |
| 2024-12-31 23:27:51,310 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:27:51,310 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json with policy now |
| 2024-12-31 23:27:51,310 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:27:51,310 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,311 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,311 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:27:51,311 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,315 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,315 DEBUG SenderThread:1764142 [sender.py:send():379] send: telemetry |
| 2024-12-31 23:27:51,315 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,316 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,316 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,316 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,316 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,317 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,317 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,317 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,317 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,317 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,318 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,318 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,318 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,318 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,319 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,319 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,319 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,319 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,320 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,320 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,320 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:51,320 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:51,320 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: log_artifact |
| 2024-12-31 23:27:51,549 INFO wandb-upload_2:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/r0q1x21y-media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:27:51,717 INFO wandb-upload_1:1764142 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmp28rcpsod |
| 2024-12-31 23:27:51,782 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:27:51,782 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:27:51,782 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table/eval/total_success_rate_ci_6_cdffd89ce3ca7a2fac89.table.json |
| 2024-12-31 23:27:51,782 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table/eval |
| 2024-12-31 23:27:52,101 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-31 23:27:52,113 INFO SenderThread:1764142 [sender.py:send_request_log_artifact():1525] logged artifact run-ix23zko1-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNDE5Nzg0OQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTE3ODA3OQ==', 'latestArtifact': None}} |
| 2024-12-31 23:27:52,113 DEBUG SenderThread:1764142 [sender.py:send():379] send: files |
| 2024-12-31 23:27:52,113 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_cdffd89ce3ca7a2fac89.table.json with policy now |
| 2024-12-31 23:27:52,115 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:27:52,115 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:27:52,115 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:52,115 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:27:52,116 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-31 23:27:52,243 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:27:52,280 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:52,308 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/r637fazs-media/table/eval/total_success_rate_ci_6_cdffd89ce3ca7a2fac89.table.json |
| 2024-12-31 23:27:52,567 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/s183w4yz-media/table/eval/total_success_rate_ci_6_cdffd89ce3ca7a2fac89.table.json |
| 2024-12-31 23:27:52,782 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:27:52,782 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:27:53,280 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:54,280 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:27:54,593 DEBUG SenderThread:1764142 [sender.py:send():379] send: exit |
| 2024-12-31 23:27:54,594 INFO SenderThread:1764142 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-31 23:27:54,594 INFO SenderThread:1764142 [sender.py:send_exit():588] handling runtime: 302 |
| 2024-12-31 23:27:54,594 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:27:54,594 INFO SenderThread:1764142 [sender.py:send_exit():594] send defer |
| 2024-12-31 23:27:54,594 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:27:54,594 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-31 23:27:54,595 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:27:54,595 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-31 23:27:54,595 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-31 23:27:54,595 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:27:54,595 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-31 23:27:54,595 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:27:54,595 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-31 23:27:54,595 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-31 23:27:54,595 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:27:54,595 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-31 23:27:54,595 INFO HandlerThread:1764142 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-31 23:27:54,595 DEBUG SystemMonitor:1764142 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-31 23:27:54,595 DEBUG SystemMonitor:1764142 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-31 23:27:54,596 INFO HandlerThread:1764142 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-31 23:27:54,596 INFO HandlerThread:1764142 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-31 23:27:54,650 INFO HandlerThread:1764142 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-31 23:27:54,650 INFO HandlerThread:1764142 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-31 23:27:54,650 INFO HandlerThread:1764142 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-31 23:27:54,651 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:27:54,651 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-31 23:27:54,651 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-31 23:27:54,651 DEBUG SenderThread:1764142 [sender.py:send():379] send: stats |
| 2024-12-31 23:27:54,651 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:27:54,651 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-31 23:27:54,651 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:27:54,651 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-31 23:27:54,651 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-31 23:27:54,652 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:27:54,652 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-31 23:27:54,783 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:27:56,784 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:28:01,525 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:28:01,525 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:28:01,525 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-31 23:28:01,526 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:01,526 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-31 23:28:01,526 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-31 23:28:01,526 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:28:01,526 DEBUG SenderThread:1764142 [sender.py:send():379] send: history |
| 2024-12-31 23:28:01,526 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-31 23:28:01,527 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:01,527 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-31 23:28:01,527 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:28:01,527 DEBUG SenderThread:1764142 [sender.py:send():379] send: summary |
| 2024-12-31 23:28:01,531 INFO SenderThread:1764142 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-31 23:28:01,531 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:01,531 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-31 23:28:01,532 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-31 23:28:01,532 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:01,532 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-31 23:28:01,532 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:01,532 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-31 23:28:01,535 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-31 23:28:01,595 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:28:01,824 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:28:01,824 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-31 23:28:01,825 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:28:01,825 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:01,825 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-31 23:28:01,825 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:01,825 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-31 23:28:02,595 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:28:02,776 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-31 23:28:02,776 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:28:02,776 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:02,776 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-31 23:28:02,777 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:02,777 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-31 23:28:02,777 INFO SenderThread:1764142 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-31 23:28:02,779 INFO SenderThread:1764142 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-31 23:28:02,781 INFO SenderThread:1764142 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-31 23:28:02,800 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-31 23:28:02,800 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:02,800 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-31 23:28:02,800 DEBUG SenderThread:1764142 [sender.py:send():379] send: artifact |
| 2024-12-31 23:28:02,825 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/config.yaml |
| 2024-12-31 23:28:02,825 INFO Thread-12 :1764142 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:28:03,152 INFO wandb-upload_2:1764142 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmp88oy__h_ |
| 2024-12-31 23:28:03,152 INFO wandb-upload_1:1764142 [upload_job.py:push():85] Skipped uploading /tmp/tmpchtkkcxn/wandb-job.json |
| 2024-12-31 23:28:03,250 INFO wandb-upload_0:1764142 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpaxguhh2y |
| 2024-12-31 23:28:03,596 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:28:03,805 INFO SenderThread:1764142 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNDE5ODAyNA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNDE2NzY3NQ==', 'versionIndex': 398}}} |
| 2024-12-31 23:28:03,805 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:03,805 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-31 23:28:03,805 INFO SenderThread:1764142 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-31 23:28:03,825 INFO SenderThread:1764142 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/config.yaml config.yaml |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json wandb-summary.json |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/requirements.txt requirements.txt |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log output.log |
| 2024-12-31 23:28:03,826 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/diff.patch diff.patch |
| 2024-12-31 23:28:03,828 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-31 23:28:03,828 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/4_eval_4_e48861ffe574038dd2b6.mp4 media/videos/eval/4_eval_4_e48861ffe574038dd2b6.mp4 |
| 2024-12-31 23:28:03,829 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/0_eval_0_eeec50fdf1118de79305.mp4 media/videos/eval/0_eval_0_eeec50fdf1118de79305.mp4 |
| 2024-12-31 23:28:03,829 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/3_eval_3_4b8025834b9f25779448.mp4 media/videos/eval/3_eval_3_4b8025834b9f25779448.mp4 |
| 2024-12-31 23:28:03,829 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/2_eval_2_3ca0a8506093efa659f9.mp4 media/videos/eval/2_eval_2_3ca0a8506093efa659f9.mp4 |
| 2024-12-31 23:28:03,829 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/videos/eval/1_eval_1_00cd606e45f670b2df9c.mp4 media/videos/eval/1_eval_1_00cd606e45f670b2df9c.mp4 |
| 2024-12-31 23:28:03,833 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table/eval/total_success_rate_ci_6_cdffd89ce3ca7a2fac89.table.json media/table/eval/total_success_rate_ci_6_cdffd89ce3ca7a2fac89.table.json |
| 2024-12-31 23:28:03,833 INFO SenderThread:1764142 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json media/table/eval/success_rate_ci_5_36776135f97d857e6145.table.json |
| 2024-12-31 23:28:03,833 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-31 23:28:03,833 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:28:03,835 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:03,835 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-31 23:28:03,836 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:03,836 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-31 23:28:03,836 INFO SenderThread:1764142 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 23:28:04,018 INFO wandb-upload_2:1764142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/config.yaml |
| 2024-12-31 23:28:04,061 INFO wandb-upload_1:1764142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/wandb-summary.json |
| 2024-12-31 23:28:04,065 INFO wandb-upload_3:1764142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/requirements.txt |
| 2024-12-31 23:28:04,071 INFO wandb-upload_4:1764142 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/files/output.log |
| 2024-12-31 23:28:04,075 INFO wandb-upload_0:1764142 [upload_job.py:push():130] Uploaded file /tmp/tmpvim0lr5zwandb/4ixgafw6-events.out.tfevents.1735705372.leonmkim-ROG-Strix-G15CS-G15CS.1763844.0 |
| 2024-12-31 23:28:04,275 INFO Thread-11 (_thread_body):1764142 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-31 23:28:04,275 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:04,275 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-31 23:28:04,276 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:04,276 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-31 23:28:04,276 INFO SenderThread:1764142 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-31 23:28:04,277 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-31 23:28:04,277 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:04,277 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-31 23:28:04,277 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:04,277 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-31 23:28:04,277 INFO SenderThread:1764142 [file_stream.py:finish():601] file stream finish called |
| 2024-12-31 23:28:04,502 INFO SenderThread:1764142 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-31 23:28:04,502 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-31 23:28:04,502 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:04,502 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-31 23:28:04,502 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:04,502 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-31 23:28:04,502 INFO SenderThread:1764142 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-31 23:28:04,503 DEBUG SenderThread:1764142 [sender.py:send():379] send: final |
| 2024-12-31 23:28:04,503 DEBUG SenderThread:1764142 [sender.py:send():379] send: footer |
| 2024-12-31 23:28:04,503 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-31 23:28:04,503 INFO HandlerThread:1764142 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-31 23:28:04,503 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: defer |
| 2024-12-31 23:28:04,503 INFO SenderThread:1764142 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-31 23:28:04,503 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:28:04,504 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:28:04,504 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-31 23:28:04,504 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: server_info |
| 2024-12-31 23:28:04,505 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-31 23:28:04,506 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-31 23:28:04,506 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-31 23:28:04,507 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-31 23:28:04,560 DEBUG SenderThread:1764142 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-31 23:28:04,560 INFO MainThread:1764142 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-31 23:28:04,560 INFO MainThread:1764142 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-31 23:28:04,561 INFO MainThread:1764142 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-31 23:28:04,561 DEBUG HandlerThread:1764142 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-31 23:28:04,561 INFO HandlerThread:1764142 [handler.py:finish():882] shutting down handler |
| 2024-12-31 23:28:05,506 INFO WriterThread:1764142 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1007_0/ix23zko1/wandb/run-20241231_232251-ix23zko1/run-ix23zko1.wandb |
| 2024-12-31 23:28:05,560 INFO SenderThread:1764142 [sender.py:finish():1615] shutting down sender |
| 2024-12-31 23:28:05,560 INFO SenderThread:1764142 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-31 23:28:05,560 INFO SenderThread:1764142 [file_pusher.py:join():175] waiting for file pusher |
|
|
