| 2024-12-11 11:36:44,620 INFO StreamThr :743447 [internal.py:wandb_internal():85] W&B internal server running at pid: 743447, started at: 2024-12-11 11:36:44.620258 |
| 2024-12-11 11:36:44,621 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status |
| 2024-12-11 11:36:44,632 INFO WriterThread:743447 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/run-ahosb0lo.wandb |
| 2024-12-11 11:36:44,633 DEBUG SenderThread:743447 [sender.py:send():379] send: header |
| 2024-12-11 11:36:44,642 DEBUG SenderThread:743447 [sender.py:send():379] send: run |
| 2024-12-11 11:36:44,901 INFO SenderThread:743447 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files |
| 2024-12-11 11:36:44,901 INFO SenderThread:743447 [sender.py:_start_run_threads():1188] run started: ahosb0lo with start time 1733935004.620625 |
| 2024-12-11 11:36:44,921 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-11 11:36:44,921 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: check_version |
| 2024-12-11 11:36:44,995 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-11 11:36:45,000 DEBUG HandlerThread:743447 [system_info.py:__init__():26] System info init |
| 2024-12-11 11:36:45,000 DEBUG HandlerThread:743447 [system_info.py:__init__():41] System info init done |
| 2024-12-11 11:36:45,000 INFO HandlerThread:743447 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-11 11:36:45,000 INFO SystemMonitor:743447 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-11 11:36:45,000 INFO HandlerThread:743447 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-11 11:36:45,000 INFO SystemMonitor:743447 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-11 11:36:45,001 INFO SystemMonitor:743447 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-11 11:36:45,002 INFO SystemMonitor:743447 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-11 11:36:45,002 INFO SystemMonitor:743447 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-11 11:36:45,003 INFO SystemMonitor:743447 [interfaces.py:start():188] Started network monitoring |
| 2024-12-11 11:36:45,012 DEBUG HandlerThread:743447 [system_info.py:probe():152] Probing system |
| 2024-12-11 11:36:45,013 DEBUG HandlerThread:743447 [system_info.py:_probe_git():137] Probing git |
| 2024-12-11 11:36:45,018 DEBUG HandlerThread:743447 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-11 11:36:45,018 DEBUG HandlerThread:743447 [system_info.py:probe():200] Probing system done |
| 2024-12-11 11:36:45,018 DEBUG HandlerThread:743447 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-11T16:36:45.012114', 'startedAt': '2024-12-11T16:36:44.613038', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '1b557f164c7f62a8d1545c761b13ede1677c3a83'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3937.48675, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4500.011, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 661.6516265869141}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71596908569336}} |
| 2024-12-11 11:36:45,018 INFO HandlerThread:743447 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-11 11:36:45,018 INFO HandlerThread:743447 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-11 11:36:45,018 DEBUG HandlerThread:743447 [system_info.py:_save_code():44] Saving code |
| 2024-12-11 11:36:45,020 DEBUG HandlerThread:743447 [system_info.py:_save_code():67] Saving code done |
| 2024-12-11 11:36:45,020 DEBUG HandlerThread:743447 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-11 11:36:45,082 DEBUG HandlerThread:743447 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-11 11:36:45,082 INFO HandlerThread:743447 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-11 11:36:45,086 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:36:45,086 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-11 11:36:45,086 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-11 11:36:45,086 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-11 11:36:45,328 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-11 11:36:45,328 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:36:45,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-11 11:36:45,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:36:45,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:45,331 INFO HandlerThread:743447 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/tb" |
| save: true |
| } |
| control { |
| relay_id: "3b814f0f1e304546b568a989da857bea" |
| } |
| _info { |
| stream_id: "ahosb0lo" |
| } |
|
|
| 2024-12-11 11:36:45,331 DEBUG HandlerThread:743447 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-11 11:36:45,344 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/jpxgwjga-wandb-metadata.json |
| 2024-12-11 11:36:45,381 INFO wandb-upload_1:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/xzmf7vax-code/FISH/eval_robot.py |
| 2024-12-11 11:36:45,383 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/am9eq8ny-diff.patch |
| 2024-12-11 11:36:45,439 DEBUG SenderThread:743447 [sender.py:send():379] send: telemetry |
| 2024-12-11 11:36:45,439 DEBUG SenderThread:743447 [sender.py:send():379] send: tbrecord |
| 2024-12-11 11:36:45,439 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:36:45,439 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 with policy live |
| 2024-12-11 11:36:45,614 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/sdv9591m-events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:36:45,902 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-metadata.json |
| 2024-12-11 11:36:45,902 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/diff.patch |
| 2024-12-11 11:36:45,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:36:45,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/requirements.txt |
| 2024-12-11 11:36:45,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:36:45,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/code/FISH/eval_robot.py |
| 2024-12-11 11:36:45,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/code/FISH |
| 2024-12-11 11:36:45,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/code |
| 2024-12-11 11:36:46,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:47,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:47,902 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:36:48,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:49,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:49,565 DEBUG SenderThread:743447 [sender.py:send():379] send: config |
| 2024-12-11 11:36:50,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:36:50,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:51,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:51,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:36:52,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:53,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:53,903 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:36:54,346 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:55,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:36:55,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:55,904 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:36:56,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:57,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:57,905 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:36:58,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:36:59,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:00,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:37:00,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:37:00,370 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:00,483 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:01,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:02,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:03,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:04,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:05,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:06,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:06,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:07,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:08,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:09,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:10,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:11,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:11,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:12,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:13,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:14,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:15,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:37:15,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:37:15,370 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:16,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:16,419 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:16,910 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/config.yaml |
| 2024-12-11 11:37:17,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:18,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:19,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:19,911 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:20,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:21,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:21,687 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:22,334 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:23,334 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:23,912 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:24,334 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:25,334 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:25,912 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:26,334 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:26,799 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:27,336 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:27,913 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:28,336 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:29,337 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:29,913 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:30,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:37:30,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:37:30,337 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:31,337 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:31,803 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:31,914 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:32,337 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:33,338 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:33,914 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:34,337 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:35,338 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:35,915 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:36,338 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:37,047 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:37,338 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:37,916 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:38,339 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:39,339 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:39,916 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:40,339 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:41,339 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:41,917 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:42,166 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:42,339 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:43,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:43,918 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:44,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:45,003 DEBUG SystemMonitor:743447 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-11 11:37:45,004 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:37:45,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:37:45,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:37:45,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:45,918 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:46,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:47,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:47,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:47,919 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:48,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:49,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:49,919 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:50,340 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:51,341 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:51,920 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:52,341 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:52,603 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:53,341 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:53,921 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:54,341 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:55,341 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:55,921 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:56,341 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:57,288 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:37:57,289 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 11:37:57,289 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_dea3cb7ce16ad057bd5f.mp4 with policy now |
| 2024-12-11 11:37:57,290 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:37:57,290 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:37:57,290 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:37:57,342 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:57,529 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/s4xw9bmh-media/videos/eval/0_eval_0_dea3cb7ce16ad057bd5f.mp4 |
| 2024-12-11 11:37:57,922 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:37:57,922 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/0_eval_0_dea3cb7ce16ad057bd5f.mp4 |
| 2024-12-11 11:37:57,923 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:37:57,923 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos |
| 2024-12-11 11:37:57,923 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval |
| 2024-12-11 11:37:57,923 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media |
| 2024-12-11 11:37:58,202 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/7xrrwz3m-media/videos/eval/0_eval_0_dea3cb7ce16ad057bd5f.mp4 |
| 2024-12-11 11:37:58,302 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:37:58,342 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:59,342 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:37:59,923 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:00,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:38:00,329 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:38:00,342 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:01,342 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:01,923 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:02,342 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:03,345 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:03,924 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:03,962 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:04,344 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:05,344 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:06,344 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:07,344 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:07,925 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:38:08,171 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/pvc3l7fp-events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:38:08,345 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:09,345 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:09,794 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:09,926 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:10,345 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:11,346 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:12,345 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:13,346 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:13,927 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:14,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:14,851 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:15,004 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:38:15,329 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:38:15,330 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:38:15,346 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:15,928 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:16,346 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:17,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:17,929 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:18,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:19,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:19,929 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:20,073 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:20,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:21,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:21,930 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:22,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:23,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:23,931 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:24,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:25,115 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:25,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:25,931 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:26,347 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:27,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:27,932 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:28,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:29,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:29,933 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:30,147 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:30,330 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:38:30,330 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:38:30,349 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:31,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:31,934 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:32,349 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:33,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:33,934 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:34,348 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:35,171 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:35,349 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:35,935 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:36,350 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:37,350 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:37,936 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:38,350 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:39,350 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:39,936 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:40,215 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:40,352 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:41,353 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:41,937 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:42,354 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:43,354 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:43,938 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:44,354 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:45,004 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:38:45,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:38:45,331 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:38:45,354 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:45,447 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:45,938 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:46,354 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:47,357 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:47,749 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:38:47,749 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_925210ff3b36d88f3887.mp4 with policy now |
| 2024-12-11 11:38:47,750 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 11:38:47,751 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:38:47,751 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:38:47,759 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:38:47,939 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:47,939 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:38:47,940 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/1_eval_1_925210ff3b36d88f3887.mp4 |
| 2024-12-11 11:38:47,940 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval |
| 2024-12-11 11:38:47,988 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/algrlwjl-media/videos/eval/1_eval_1_925210ff3b36d88f3887.mp4 |
| 2024-12-11 11:38:48,259 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/eex9yxjq-media/videos/eval/1_eval_1_925210ff3b36d88f3887.mp4 |
| 2024-12-11 11:38:48,365 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:49,358 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:49,940 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:50,357 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:50,450 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:51,358 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:51,941 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:52,363 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:53,358 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:53,941 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:38:54,361 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:55,357 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:56,336 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:38:56,357 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:57,358 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:58,358 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:38:59,364 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:00,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:39:00,331 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:39:00,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:01,364 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:01,423 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:02,364 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:03,367 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:04,366 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:05,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:06,369 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:07,338 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:07,370 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:08,367 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:09,369 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:10,371 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:11,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:12,343 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:12,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:13,372 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:14,371 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:15,005 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:39:15,335 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:39:15,335 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:39:15,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:16,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:17,372 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:17,430 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:18,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:19,374 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:20,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:21,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:21,758 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:39:21,759 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:39:21,759 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:39:21,950 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:39:21,950 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:39:21,951 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:22,200 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/wnjvdsbf-events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:39:22,374 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:23,170 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:23,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:23,951 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:24,374 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:25,374 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:25,951 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:26,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:27,373 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:27,952 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:28,298 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:28,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:29,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:29,953 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:30,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:39:30,332 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:39:30,374 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:31,374 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:31,954 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:32,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:33,322 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:33,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:33,954 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:34,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:35,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:35,955 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:36,375 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:37,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:37,956 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:38,344 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:38,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:39,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:39,957 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:40,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:41,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:41,958 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:42,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:43,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:43,454 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:43,958 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:44,376 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:45,005 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:39:45,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:39:45,332 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:39:45,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:45,959 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:46,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:47,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:47,960 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:48,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:48,576 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:49,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:49,961 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:50,377 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:51,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:51,961 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:52,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:53,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:53,803 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:53,962 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:54,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:55,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:55,963 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:56,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:57,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:57,964 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:39:58,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:58,900 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:39:59,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:39:59,964 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:00,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:40:00,332 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:40:00,378 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:01,379 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:01,965 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:02,384 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:02,517 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 11:40:02,518 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:40:02,518 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_ddba0e3583786ac1097b.mp4 with policy now |
| 2024-12-11 11:40:02,519 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:40:02,519 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:40:02,519 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:40:02,735 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/2qt74j26-media/videos/eval/2_eval_2_ddba0e3583786ac1097b.mp4 |
| 2024-12-11 11:40:02,965 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:40:02,966 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/2_eval_2_ddba0e3583786ac1097b.mp4 |
| 2024-12-11 11:40:02,966 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval |
| 2024-12-11 11:40:03,258 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/nmnl78ir-media/videos/eval/2_eval_2_ddba0e3583786ac1097b.mp4 |
| 2024-12-11 11:40:03,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:03,966 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:04,387 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:04,529 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:05,387 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:05,966 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:06,386 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:07,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:08,386 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:09,394 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:09,533 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:09,621 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:40:09,622 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:40:09,622 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:40:09,968 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:40:09,968 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:40:09,968 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:10,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:11,387 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:11,968 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:12,398 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:13,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:13,969 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:14,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:14,562 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:15,006 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:40:15,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:40:15,332 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:40:15,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:15,970 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:16,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:17,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:17,971 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:18,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:19,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:19,788 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:19,972 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:20,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:21,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:21,973 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:22,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:23,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:23,974 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:24,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:24,792 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:25,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:25,974 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:26,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:27,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:27,975 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:28,390 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:29,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:29,860 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:29,976 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:30,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:40:30,332 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:40:30,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:31,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:31,977 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:32,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:33,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:33,977 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:34,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:35,029 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:35,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:35,978 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:36,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:37,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:37,979 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:38,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:39,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:39,979 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:40,086 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:40,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:41,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:41,980 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:42,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:43,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:43,981 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:44,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:45,014 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:40:45,130 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:45,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:40:45,333 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:40:45,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:45,982 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:46,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:47,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:47,983 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:48,393 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:49,396 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:49,983 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:50,091 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:40:50,091 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_b687720b2d32aeeb9a91.mp4 with policy now |
| 2024-12-11 11:40:50,094 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 11:40:50,094 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:40:50,095 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:40:50,105 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:40:50,133 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:50,329 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/6p2eraxb-media/videos/eval/3_eval_3_b687720b2d32aeeb9a91.mp4 |
| 2024-12-11 11:40:50,394 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:50,987 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:40:50,987 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/3_eval_3_b687720b2d32aeeb9a91.mp4 |
| 2024-12-11 11:40:50,987 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval |
| 2024-12-11 11:40:51,327 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/aiy14acg-media/videos/eval/3_eval_3_b687720b2d32aeeb9a91.mp4 |
| 2024-12-11 11:40:51,406 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:51,987 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:52,399 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:53,404 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:53,988 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:54,403 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:55,358 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:40:55,409 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:56,411 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:57,413 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:57,989 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:40:58,410 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:40:59,410 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:00,335 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:41:00,336 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:41:00,411 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:00,432 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:00,683 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:41:00,683 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:41:00,683 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:41:00,991 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:41:00,991 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:41:01,267 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/8cztmbl3-events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:41:01,410 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:01,991 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:02,412 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:03,414 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:04,409 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:05,446 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:05,446 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:05,992 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:06,421 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:07,421 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:07,993 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:08,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:09,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:09,994 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:10,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:10,633 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:11,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:11,994 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:12,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:13,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:13,995 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:14,422 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:15,007 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:41:15,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:41:15,334 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:41:15,423 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:15,812 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:15,996 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:16,423 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:17,424 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:17,997 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:18,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:19,426 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:19,998 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:20,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:20,844 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:21,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:21,998 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:22,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:23,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:23,999 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:24,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:25,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:25,874 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:26,000 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:26,425 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:27,426 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:28,001 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:28,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:29,430 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:30,001 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:30,333 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:41:30,334 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:41:30,430 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:31,070 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:31,430 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:32,002 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:32,430 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:33,430 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:34,003 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:34,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:35,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:36,004 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:36,123 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:36,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:37,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:38,005 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:38,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:39,431 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:40,005 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:40,433 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:40,811 DEBUG SenderThread:743447 [sender.py:send():379] send: files |
| 2024-12-11 11:41:40,811 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_10441113b4d3ea44e8c6.mp4 with policy now |
| 2024-12-11 11:41:40,812 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-11 11:41:40,812 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:41:40,812 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:41:40,820 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:41:41,006 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:41:41,006 INFO Thread-12 :743447 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/4_eval_4_10441113b4d3ea44e8c6.mp4 |
| 2024-12-11 11:41:41,006 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval |
| 2024-12-11 11:41:41,043 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/0somhq86-media/videos/eval/4_eval_4_10441113b4d3ea44e8c6.mp4 |
| 2024-12-11 11:41:41,317 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/56mpvpsh-media/videos/eval/4_eval_4_10441113b4d3ea44e8c6.mp4 |
| 2024-12-11 11:41:41,437 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:41,867 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:42,006 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:42,441 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:43,435 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:44,007 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:44,439 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:45,007 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:41:45,345 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-11 11:41:45,345 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-11 11:41:45,442 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:46,441 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:47,446 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:47,542 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:48,009 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:48,446 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:49,450 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:50,455 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:41:51,330 DEBUG SenderThread:743447 [sender.py:send():379] send: exit |
| 2024-12-11 11:41:51,330 INFO SenderThread:743447 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-11 11:41:51,330 INFO SenderThread:743447 [sender.py:send_exit():588] handling runtime: 306 |
| 2024-12-11 11:41:51,330 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:41:51,330 INFO SenderThread:743447 [sender.py:send_exit():594] send defer |
| 2024-12-11 11:41:51,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:51,331 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-11 11:41:51,331 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:51,331 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-11 11:41:51,331 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-11 11:41:51,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:51,331 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-11 11:41:51,331 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:51,331 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-11 11:41:51,331 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-11 11:41:51,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:51,332 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-11 11:41:51,332 INFO HandlerThread:743447 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-11 11:41:51,332 DEBUG SystemMonitor:743447 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-11 11:41:51,332 INFO HandlerThread:743447 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-11 11:41:51,332 DEBUG SystemMonitor:743447 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-11 11:41:51,332 INFO HandlerThread:743447 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-11 11:41:51,380 INFO HandlerThread:743447 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-11 11:41:51,381 INFO HandlerThread:743447 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-11 11:41:51,381 INFO HandlerThread:743447 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-11 11:41:51,381 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:51,381 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-11 11:41:51,381 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-11 11:41:51,381 DEBUG SenderThread:743447 [sender.py:send():379] send: stats |
| 2024-12-11 11:41:51,381 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:51,382 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-11 11:41:51,382 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:51,382 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-11 11:41:51,382 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-11 11:41:51,382 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:51,382 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-11 11:41:52,010 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:41:52,011 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:41:53,011 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:57,384 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 11:41:57,384 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:57,384 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:57,388 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-11 11:41:57,388 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-11 11:41:57,388 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-11 11:41:57,389 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:41:57,389 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:57,389 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:41:57,389 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-11 11:41:57,389 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:41:57,389 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 11:41:57,390 DEBUG SenderThread:743447 [sender.py:send():379] send: history |
| 2024-12-11 11:41:57,390 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-11 11:41:57,390 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:41:57,390 DEBUG SenderThread:743447 [sender.py:send():379] send: summary |
| 2024-12-11 11:41:57,391 INFO SenderThread:743447 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-11 11:41:57,391 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:57,391 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-11 11:41:57,391 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-11 11:41:57,391 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:57,391 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-11 11:41:57,391 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:57,392 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-11 11:41:57,392 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-11 11:41:57,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-11 11:41:57,392 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:57,392 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-11 11:41:57,392 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:57,392 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-11 11:41:58,012 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:41:58,015 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-11 11:41:58,015 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:58,015 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-11 11:41:58,015 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:58,015 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-11 11:41:58,015 INFO SenderThread:743447 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-11 11:41:58,017 INFO SenderThread:743447 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-11 11:41:58,032 INFO SenderThread:743447 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-11 11:41:58,046 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-11 11:41:58,046 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:41:58,047 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-11 11:41:58,047 DEBUG SenderThread:743447 [sender.py:send():379] send: artifact |
| 2024-12-11 11:41:58,331 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 11:41:58,562 INFO wandb-upload_1:743447 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpe19lto7x |
| 2024-12-11 11:41:58,562 INFO wandb-upload_2:743447 [upload_job.py:push():85] Skipped uploading /tmp/tmpybw54_lu/wandb-job.json |
| 2024-12-11 11:41:58,687 INFO wandb-upload_0:743447 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpugla6_vr |
| 2024-12-11 11:41:59,013 INFO Thread-12 :743447 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:41:59,244 INFO SenderThread:743447 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM2NzYwNTg3Nw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM2NzYwMDc2Nw==', 'versionIndex': 364}}} |
| 2024-12-11 11:41:59,245 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:41:59,245 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-11 11:41:59,245 INFO SenderThread:743447 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-11 11:42:00,013 INFO SenderThread:743447 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files |
| 2024-12-11 11:42:00,013 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/config.yaml config.yaml |
| 2024-12-11 11:42:00,014 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:42:00,015 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json wandb-summary.json |
| 2024-12-11 11:42:00,015 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/requirements.txt requirements.txt |
| 2024-12-11 11:42:00,016 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-11 11:42:00,016 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log output.log |
| 2024-12-11 11:42:00,016 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/diff.patch diff.patch |
| 2024-12-11 11:42:00,016 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-11 11:42:00,022 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/0_eval_0_dea3cb7ce16ad057bd5f.mp4 media/videos/eval/0_eval_0_dea3cb7ce16ad057bd5f.mp4 |
| 2024-12-11 11:42:00,022 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/4_eval_4_10441113b4d3ea44e8c6.mp4 media/videos/eval/4_eval_4_10441113b4d3ea44e8c6.mp4 |
| 2024-12-11 11:42:00,022 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/2_eval_2_ddba0e3583786ac1097b.mp4 media/videos/eval/2_eval_2_ddba0e3583786ac1097b.mp4 |
| 2024-12-11 11:42:00,022 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/3_eval_3_b687720b2d32aeeb9a91.mp4 media/videos/eval/3_eval_3_b687720b2d32aeeb9a91.mp4 |
| 2024-12-11 11:42:00,022 INFO SenderThread:743447 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/media/videos/eval/1_eval_1_925210ff3b36d88f3887.mp4 media/videos/eval/1_eval_1_925210ff3b36d88f3887.mp4 |
| 2024-12-11 11:42:00,022 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-11 11:42:00,023 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 11:42:00,023 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:42:00,023 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-11 11:42:00,023 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:42:00,023 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-11 11:42:00,023 INFO SenderThread:743447 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-11 11:42:00,215 INFO wandb-upload_1:743447 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/config.yaml |
| 2024-12-11 11:42:00,233 INFO wandb-upload_2:743447 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/wandb-summary.json |
| 2024-12-11 11:42:00,247 INFO wandb-upload_3:743447 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/requirements.txt |
| 2024-12-11 11:42:00,264 INFO wandb-upload_4:743447 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/files/output.log |
| 2024-12-11 11:42:00,332 INFO wandb-upload_0:743447 [upload_job.py:push():130] Uploaded file /tmp/tmp0lcadrfewandb/o5djqyqe-events.out.tfevents.1733935005.leonmkim-ROG-Strix-G15CS-G15CS.743294.0 |
| 2024-12-11 11:42:00,332 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 11:42:00,333 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 11:42:00,532 INFO Thread-11 (_thread_body):743447 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-11 11:42:00,533 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:42:00,533 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-11 11:42:00,533 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:42:00,533 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-11 11:42:00,533 INFO SenderThread:743447 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-11 11:42:00,534 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-11 11:42:00,534 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:42:00,534 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-11 11:42:00,534 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:42:00,534 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-11 11:42:00,534 INFO SenderThread:743447 [file_stream.py:finish():601] file stream finish called |
| 2024-12-11 11:42:00,730 INFO SenderThread:743447 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-11 11:42:00,731 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-11 11:42:00,731 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:42:00,731 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-11 11:42:00,731 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:42:00,731 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-11 11:42:00,731 INFO SenderThread:743447 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-11 11:42:00,731 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-11 11:42:00,731 DEBUG SenderThread:743447 [sender.py:send():379] send: final |
| 2024-12-11 11:42:00,731 INFO HandlerThread:743447 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-11 11:42:00,731 DEBUG SenderThread:743447 [sender.py:send():379] send: footer |
| 2024-12-11 11:42:00,731 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: defer |
| 2024-12-11 11:42:00,731 INFO SenderThread:743447 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-11 11:42:00,732 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 11:42:00,732 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 11:42:00,732 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-11 11:42:00,732 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: server_info |
| 2024-12-11 11:42:00,734 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-11 11:42:00,734 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-11 11:42:00,734 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-11 11:42:00,735 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-11 11:42:00,795 DEBUG SenderThread:743447 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-11 11:42:00,795 INFO MainThread:743447 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-11 11:42:00,796 INFO MainThread:743447 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-11 11:42:00,796 INFO MainThread:743447 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-11 11:42:00,796 DEBUG HandlerThread:743447 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-11 11:42:00,796 INFO HandlerThread:743447 [handler.py:finish():882] shutting down handler |
| 2024-12-11 11:42:01,734 INFO WriterThread:743447 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/265650/113638/wandb/run-20241211_113644-ahosb0lo/run-ahosb0lo.wandb |
| 2024-12-11 11:42:01,795 INFO SenderThread:743447 [sender.py:finish():1615] shutting down sender |
| 2024-12-11 11:42:01,795 INFO SenderThread:743447 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-11 11:42:01,795 INFO SenderThread:743447 [file_pusher.py:join():175] waiting for file pusher |
|
|
