| 2024-12-17 21:52:46,130 INFO StreamThr :78001 [internal.py:wandb_internal():85] W&B internal server running at pid: 78001, started at: 2024-12-17 21:52:46.129837 |
| 2024-12-17 21:52:46,131 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status |
| 2024-12-17 21:52:46,144 INFO WriterThread:78001 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/run-g6nsr00w.wandb |
| 2024-12-17 21:52:46,144 DEBUG SenderThread:78001 [sender.py:send():379] send: header |
| 2024-12-17 21:52:46,152 DEBUG SenderThread:78001 [sender.py:send():379] send: run |
| 2024-12-17 21:52:46,475 INFO SenderThread:78001 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files |
| 2024-12-17 21:52:46,475 INFO SenderThread:78001 [sender.py:_start_run_threads():1188] run started: g6nsr00w with start time 1734490366.130536 |
| 2024-12-17 21:52:46,494 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: check_version |
| 2024-12-17 21:52:46,494 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: check_version |
| 2024-12-17 21:52:46,540 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: run_start |
| 2024-12-17 21:52:46,544 DEBUG HandlerThread:78001 [system_info.py:__init__():26] System info init |
| 2024-12-17 21:52:46,544 DEBUG HandlerThread:78001 [system_info.py:__init__():41] System info init done |
| 2024-12-17 21:52:46,544 INFO HandlerThread:78001 [system_monitor.py:start():194] Starting system monitor |
| 2024-12-17 21:52:46,545 INFO SystemMonitor:78001 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2024-12-17 21:52:46,545 INFO HandlerThread:78001 [system_monitor.py:probe():214] Collecting system info |
| 2024-12-17 21:52:46,545 INFO SystemMonitor:78001 [interfaces.py:start():188] Started cpu monitoring |
| 2024-12-17 21:52:46,546 INFO SystemMonitor:78001 [interfaces.py:start():188] Started disk monitoring |
| 2024-12-17 21:52:46,546 INFO SystemMonitor:78001 [interfaces.py:start():188] Started gpu monitoring |
| 2024-12-17 21:52:46,547 INFO SystemMonitor:78001 [interfaces.py:start():188] Started memory monitoring |
| 2024-12-17 21:52:46,547 INFO SystemMonitor:78001 [interfaces.py:start():188] Started network monitoring |
| 2024-12-17 21:52:46,558 DEBUG HandlerThread:78001 [system_info.py:probe():152] Probing system |
| 2024-12-17 21:52:46,560 DEBUG HandlerThread:78001 [system_info.py:_probe_git():137] Probing git |
| 2024-12-17 21:52:46,564 DEBUG HandlerThread:78001 [system_info.py:_probe_git():145] Probing git done |
| 2024-12-17 21:52:46,564 DEBUG HandlerThread:78001 [system_info.py:probe():200] Probing system done |
| 2024-12-17 21:52:46,564 DEBUG HandlerThread:78001 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2024-12-18T02:52:46.558919', 'startedAt': '2024-12-18T02:52:46.119968', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': '5f28b54373d7d41e4ebace380ff1080a8ffc4051'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3572.842125, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4517.241, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4536.345, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 392.8499336242676}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2024-12-17 21:52:46,565 INFO HandlerThread:78001 [system_monitor.py:probe():224] Finished collecting system info |
| 2024-12-17 21:52:46,565 INFO HandlerThread:78001 [system_monitor.py:probe():227] Publishing system info |
| 2024-12-17 21:52:46,565 DEBUG HandlerThread:78001 [system_info.py:_save_code():44] Saving code |
| 2024-12-17 21:52:46,567 DEBUG HandlerThread:78001 [system_info.py:_save_code():67] Saving code done |
| 2024-12-17 21:52:46,567 DEBUG HandlerThread:78001 [system_info.py:_save_patches():84] Saving git patches |
| 2024-12-17 21:52:46,676 DEBUG HandlerThread:78001 [system_info.py:_save_patches():126] Saving git patches done |
| 2024-12-17 21:52:46,677 INFO HandlerThread:78001 [system_monitor.py:probe():229] Finished publishing system info |
| 2024-12-17 21:52:46,680 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:52:46,680 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2024-12-17 21:52:46,680 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2024-12-17 21:52:46,680 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2024-12-17 21:52:46,907 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: python_packages |
| 2024-12-17 21:52:46,907 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: python_packages |
| 2024-12-17 21:52:46,908 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:52:46,908 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:52:46,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:46,910 INFO HandlerThread:78001 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/tb" |
| save: true |
| } |
| control { |
| relay_id: "394677d9b6aa4385a9b0ebca7a0e61ef" |
| } |
| _info { |
| stream_id: "g6nsr00w" |
| } |
|
|
| 2024-12-17 21:52:46,911 DEBUG HandlerThread:78001 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2024-12-17 21:52:46,935 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/cmvh8ay6-wandb-metadata.json |
| 2024-12-17 21:52:46,944 INFO wandb-upload_1:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/iupp8rxj-code/FISH/eval_robot.py |
| 2024-12-17 21:52:46,953 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/fmlykv68-diff.patch |
| 2024-12-17 21:52:47,056 DEBUG SenderThread:78001 [sender.py:send():379] send: telemetry |
| 2024-12-17 21:52:47,056 DEBUG SenderThread:78001 [sender.py:send():379] send: tbrecord |
| 2024-12-17 21:52:47,057 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:52:47,057 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 with policy live |
| 2024-12-17 21:52:47,234 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/xsh0mhuy-events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-metadata.json |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/requirements.txt |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/code/FISH/eval_robot.py |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/diff.patch |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:52:47,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/code |
| 2024-12-17 21:52:47,478 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/code/FISH |
| 2024-12-17 21:52:47,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:48,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:49,476 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:52:49,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:50,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:51,908 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:52:51,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:52,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:53,720 DEBUG SenderThread:78001 [sender.py:send():379] send: config |
| 2024-12-17 21:52:53,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:54,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:55,477 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:52:55,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:56,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:52:56,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:57,478 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:52:57,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:58,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:52:59,479 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:52:59,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:00,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:01,908 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:53:01,908 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:53:01,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:02,060 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:02,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:03,480 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:03,910 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:04,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:05,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:06,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:07,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:07,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:08,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:09,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:10,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:11,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:12,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:12,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:13,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:14,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:15,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:16,908 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:53:16,908 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:53:16,949 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:17,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:18,033 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:18,486 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/config.yaml |
| 2024-12-17 21:53:18,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:19,913 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:20,913 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:21,487 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:21,913 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:22,913 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:23,488 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:23,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:23,913 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:24,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:25,488 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:25,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:26,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:27,489 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:27,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:28,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:29,014 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:29,490 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:29,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:30,914 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:31,490 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:31,908 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:53:31,908 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:53:31,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:32,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:33,491 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:33,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:34,165 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:34,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:35,491 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:35,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:36,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:37,492 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:37,915 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:38,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:39,330 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:39,492 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:39,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:40,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:41,493 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:41,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:42,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:43,494 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:43,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:44,529 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:44,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:45,494 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:45,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:46,548 DEBUG SystemMonitor:78001 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2024-12-17 21:53:46,548 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:53:46,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:53:46,909 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:53:46,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:47,495 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:47,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:48,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:49,495 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:49,548 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:49,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:50,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:51,496 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:51,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:52,319 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:53:52,319 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_dbd1eabcbea39a449528.mp4 with policy now |
| 2024-12-17 21:53:52,319 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:53:52,320 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:53:52,320 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:53:52,320 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:53:52,496 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:53:52,497 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/0_eval_0_dbd1eabcbea39a449528.mp4 |
| 2024-12-17 21:53:52,497 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval |
| 2024-12-17 21:53:52,497 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos |
| 2024-12-17 21:53:52,497 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media |
| 2024-12-17 21:53:52,542 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/rdsdzo9c-media/videos/eval/0_eval_0_dbd1eabcbea39a449528.mp4 |
| 2024-12-17 21:53:52,775 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/ydvne3ig-media/videos/eval/0_eval_0_dbd1eabcbea39a449528.mp4 |
| 2024-12-17 21:53:52,918 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:53,497 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:53,918 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:54,918 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:55,331 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:53:55,918 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:56,918 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:57,498 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:57,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:58,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:53:59,499 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:53:59,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:00,689 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:00,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:01,500 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:54:01,741 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/kizcyfj9-events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:54:01,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:54:01,909 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:54:01,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:02,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:03,500 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:03,919 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:04,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:05,834 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:05,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:06,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:07,502 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:07,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:08,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:09,502 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:09,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:10,869 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:10,920 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:11,503 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:11,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:12,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:13,504 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:13,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:14,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:15,505 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:15,871 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:15,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:16,549 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:54:16,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:54:16,909 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:54:16,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:17,505 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:17,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:18,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:19,506 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:19,923 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:20,871 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:20,922 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:21,507 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:21,922 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:22,923 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:23,507 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:23,923 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:24,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:25,508 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:25,924 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:26,078 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:26,924 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:27,509 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:27,924 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:28,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:29,510 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:29,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:30,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:31,274 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:31,510 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:31,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:54:31,909 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:54:31,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:32,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:33,409 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:54:33,409 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_42cf14600b8d2071849c.mp4 with policy now |
| 2024-12-17 21:54:33,412 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:54:33,412 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:54:33,413 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:54:33,413 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:54:33,511 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:33,511 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:54:33,511 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/1_eval_1_42cf14600b8d2071849c.mp4 |
| 2024-12-17 21:54:33,512 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval |
| 2024-12-17 21:54:33,641 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/hvxa2wmt-media/videos/eval/1_eval_1_42cf14600b8d2071849c.mp4 |
| 2024-12-17 21:54:33,894 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/hllcqyuf-media/videos/eval/1_eval_1_42cf14600b8d2071849c.mp4 |
| 2024-12-17 21:54:33,930 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:34,931 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:35,512 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:35,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:36,427 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:36,933 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:37,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:38,933 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:39,513 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:39,931 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:40,931 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:41,514 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:41,861 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:41,932 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:42,932 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:43,932 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:44,932 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:45,932 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:46,549 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:54:46,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:54:46,909 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:54:46,934 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:47,040 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:47,312 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:54:47,312 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:54:47,312 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:54:47,516 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:47,516 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:54:47,516 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:54:47,934 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:48,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:49,517 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:49,934 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:50,934 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:51,517 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:51,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:52,197 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:52,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:53,518 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:53,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:54,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:55,519 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:55,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:56,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:57,220 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:54:57,520 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:57,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:58,935 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:54:59,520 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:54:59,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:00,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:01,521 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:01,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:55:01,909 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:55:01,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:02,227 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:02,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:03,522 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:03,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:04,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:05,523 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:05,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:06,936 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:07,423 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:07,523 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:07,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:08,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:09,524 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:09,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:10,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:11,525 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:11,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:12,610 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:12,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:13,525 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:13,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:14,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:15,526 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:15,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:16,550 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:55:16,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:55:16,910 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:55:16,937 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:17,529 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:17,610 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:17,942 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:18,608 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:55:18,608 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_5329c53c34465295af0a.mp4 with policy now |
| 2024-12-17 21:55:18,609 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:55:18,609 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:55:18,609 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:55:18,610 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:55:18,851 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/rquo0y7d-media/videos/eval/2_eval_2_5329c53c34465295af0a.mp4 |
| 2024-12-17 21:55:18,941 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:19,529 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:19,529 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:55:19,530 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/2_eval_2_5329c53c34465295af0a.mp4 |
| 2024-12-17 21:55:19,530 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval |
| 2024-12-17 21:55:19,800 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/11petlc4-media/videos/eval/2_eval_2_5329c53c34465295af0a.mp4 |
| 2024-12-17 21:55:19,941 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:20,946 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:21,942 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:22,944 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:23,285 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:23,531 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:23,942 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:24,948 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:25,531 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:25,943 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:26,951 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:27,532 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:27,946 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:28,917 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:28,947 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:29,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:30,949 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:31,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:55:31,910 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:55:31,946 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:32,154 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:55:32,154 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:55:32,155 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:55:32,534 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:55:32,534 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:55:32,777 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/2vpbc8if-events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:55:32,954 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:33,534 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:33,949 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:34,645 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:34,948 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:35,949 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:36,947 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:37,535 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:37,947 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:38,947 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:39,536 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:39,665 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:39,948 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:40,948 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:41,537 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:41,948 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:42,949 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:43,537 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:43,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:44,867 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:44,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:45,538 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:45,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:46,550 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:55:46,909 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:55:46,910 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:55:46,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:47,539 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:47,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:48,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:49,540 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:49,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:50,059 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:50,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:51,541 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:51,950 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:52,951 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:53,541 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:53,951 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:54,951 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:55,256 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:55:55,542 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:55,955 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:56,842 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:55:56,842 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_f7f27b273940874a00d0.mp4 with policy now |
| 2024-12-17 21:55:56,847 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:55:56,847 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:55:56,847 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:55:56,848 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:55:56,955 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:57,057 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/i1alqjlt-media/videos/eval/3_eval_3_f7f27b273940874a00d0.mp4 |
| 2024-12-17 21:55:57,543 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:55:57,543 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:55:57,543 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/3_eval_3_f7f27b273940874a00d0.mp4 |
| 2024-12-17 21:55:57,543 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval |
| 2024-12-17 21:55:57,793 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/zvcyis0o-media/videos/eval/3_eval_3_f7f27b273940874a00d0.mp4 |
| 2024-12-17 21:55:57,955 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:58,966 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:55:59,967 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:00,295 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:00,968 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:01,544 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:01,912 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:56:01,912 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:56:01,971 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:02,965 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:03,545 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:03,969 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:04,971 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:05,977 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:06,157 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:06,972 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:07,974 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:08,982 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:09,976 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:10,980 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:11,925 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:11,983 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:12,984 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:13,986 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:14,984 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:15,984 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:16,550 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:56:16,916 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:56:16,916 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:56:16,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:17,034 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:17,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:18,550 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:56:18,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:19,211 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:56:19,211 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:56:19,212 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:56:19,550 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:19,550 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:56:19,985 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:20,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:21,551 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:21,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:22,275 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:22,986 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:23,552 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:23,987 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:24,987 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:25,552 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:25,987 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:26,987 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:27,334 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:27,553 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:27,987 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:28,987 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:29,554 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:29,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:30,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:31,555 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:31,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:56:31,911 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:56:31,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:32,337 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:32,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:33,556 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:33,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:34,988 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:35,556 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:35,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:36,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:37,525 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:37,557 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:37,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:38,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:39,561 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:39,989 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:40,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:41,566 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:41,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:42,525 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:42,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:43,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:44,568 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:44,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:45,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:46,551 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:56:46,569 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:46,911 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:56:46,911 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:56:46,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:47,527 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:47,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:48,570 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:48,990 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:49,995 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:50,434 DEBUG SenderThread:78001 [sender.py:send():379] send: files |
| 2024-12-17 21:56:50,435 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_08fea80871bf35a0fb30.mp4 with policy now |
| 2024-12-17 21:56:50,437 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: partial_history |
| 2024-12-17 21:56:50,437 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:56:50,438 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:56:50,438 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:56:50,571 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:50,571 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:56:50,571 INFO Thread-12 :78001 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/4_eval_4_08fea80871bf35a0fb30.mp4 |
| 2024-12-17 21:56:50,572 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval |
| 2024-12-17 21:56:50,640 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/rw2u2d5b-media/videos/eval/4_eval_4_08fea80871bf35a0fb30.mp4 |
| 2024-12-17 21:56:50,894 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/z1u4c5tn-media/videos/eval/4_eval_4_08fea80871bf35a0fb30.mp4 |
| 2024-12-17 21:56:51,006 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:52,023 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:52,571 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:52,929 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:52,999 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:54,021 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:55,000 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:56,013 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:56,573 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:57,001 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:57,973 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:56:58,012 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:56:58,573 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:56:59,014 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:00,014 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:01,017 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:01,921 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: stop_status |
| 2024-12-17 21:57:01,921 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: stop_status |
| 2024-12-17 21:57:02,015 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:03,016 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:03,017 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:57:04,016 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:04,437 DEBUG SenderThread:78001 [sender.py:send():379] send: exit |
| 2024-12-17 21:57:04,437 INFO SenderThread:78001 [sender.py:send_exit():586] handling exit code: 0 |
| 2024-12-17 21:57:04,437 INFO SenderThread:78001 [sender.py:send_exit():588] handling runtime: 257 |
| 2024-12-17 21:57:04,438 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:57:04,438 INFO SenderThread:78001 [sender.py:send_exit():594] send defer |
| 2024-12-17 21:57:04,438 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:04,438 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2024-12-17 21:57:04,438 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:04,438 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2024-12-17 21:57:04,438 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 1 |
| 2024-12-17 21:57:04,438 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:04,438 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2024-12-17 21:57:04,438 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:04,438 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2024-12-17 21:57:04,438 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 2 |
| 2024-12-17 21:57:04,439 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:04,439 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2024-12-17 21:57:04,439 INFO HandlerThread:78001 [system_monitor.py:finish():203] Stopping system monitor |
| 2024-12-17 21:57:04,439 DEBUG SystemMonitor:78001 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2024-12-17 21:57:04,439 INFO HandlerThread:78001 [interfaces.py:finish():200] Joined cpu monitor |
| 2024-12-17 21:57:04,439 DEBUG SystemMonitor:78001 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2024-12-17 21:57:04,439 INFO HandlerThread:78001 [interfaces.py:finish():200] Joined disk monitor |
| 2024-12-17 21:57:04,486 INFO HandlerThread:78001 [interfaces.py:finish():200] Joined gpu monitor |
| 2024-12-17 21:57:04,486 INFO HandlerThread:78001 [interfaces.py:finish():200] Joined memory monitor |
| 2024-12-17 21:57:04,487 INFO HandlerThread:78001 [interfaces.py:finish():200] Joined network monitor |
| 2024-12-17 21:57:04,487 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:04,487 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2024-12-17 21:57:04,487 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 3 |
| 2024-12-17 21:57:04,487 DEBUG SenderThread:78001 [sender.py:send():379] send: stats |
| 2024-12-17 21:57:04,487 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:04,487 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2024-12-17 21:57:04,488 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:04,488 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2024-12-17 21:57:04,488 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 4 |
| 2024-12-17 21:57:04,488 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:04,488 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2024-12-17 21:57:04,575 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:57:04,576 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:57:04,810 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /tmp/tmpbfuocajxwandb/ypj7c7k4-events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:57:06,576 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:57:10,491 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:57:10,491 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:57:10,491 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:10,491 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: keepalive |
| 2024-12-17 21:57:10,491 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2024-12-17 21:57:10,492 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 5 |
| 2024-12-17 21:57:10,492 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:57:10,492 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:57:10,492 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:10,493 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2024-12-17 21:57:10,493 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:57:10,493 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:57:10,493 DEBUG SenderThread:78001 [sender.py:send():379] send: history |
| 2024-12-17 21:57:10,494 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: summary_record |
| 2024-12-17 21:57:10,497 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:57:10,497 DEBUG SenderThread:78001 [sender.py:send():379] send: summary |
| 2024-12-17 21:57:10,497 INFO SenderThread:78001 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2024-12-17 21:57:10,497 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:10,497 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2024-12-17 21:57:10,498 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 6 |
| 2024-12-17 21:57:10,498 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:10,498 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2024-12-17 21:57:10,498 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:10,498 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2024-12-17 21:57:10,498 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 7 |
| 2024-12-17 21:57:10,498 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: status_report |
| 2024-12-17 21:57:10,498 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:10,498 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2024-12-17 21:57:10,498 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:10,498 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2024-12-17 21:57:10,577 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:57:11,439 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:57:13,429 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 8 |
| 2024-12-17 21:57:13,429 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:57:13,429 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:13,429 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2024-12-17 21:57:13,430 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:13,430 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2024-12-17 21:57:13,430 INFO SenderThread:78001 [job_builder.py:build():440] Attempting to build job artifact |
| 2024-12-17 21:57:13,432 INFO SenderThread:78001 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2024-12-17 21:57:13,439 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:57:13,451 INFO SenderThread:78001 [job_builder.py:build():545] adding wandb-job metadata file |
| 2024-12-17 21:57:13,461 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 9 |
| 2024-12-17 21:57:13,462 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:57:13,462 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:13,462 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2024-12-17 21:57:13,462 DEBUG SenderThread:78001 [sender.py:send():379] send: artifact |
| 2024-12-17 21:57:13,579 INFO Thread-12 :78001 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:57:14,015 INFO wandb-upload_0:78001 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpkt9zbjlr |
| 2024-12-17 21:57:14,015 INFO wandb-upload_1:78001 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpxnpvbcq3 |
| 2024-12-17 21:57:14,229 INFO wandb-upload_2:78001 [upload_job.py:push():88] Uploaded file /tmp/tmpz2cro5k3/wandb-job.json |
| 2024-12-17 21:57:14,440 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:57:14,830 INFO SenderThread:78001 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTM4MzMzNjA0Mg==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTM4MzIyNjY3MQ==', 'versionIndex': 379}}} |
| 2024-12-17 21:57:14,830 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:14,830 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2024-12-17 21:57:14,830 INFO SenderThread:78001 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2024-12-17 21:57:15,580 INFO SenderThread:78001 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files |
| 2024-12-17 21:57:15,581 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/config.yaml config.yaml |
| 2024-12-17 21:57:15,581 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 events.out.tfevents.1734490366.leonmkim-ROG-Strix-G15CS-G15CS.77808.0 |
| 2024-12-17 21:57:15,581 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json wandb-summary.json |
| 2024-12-17 21:57:15,581 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/requirements.txt requirements.txt |
| 2024-12-17 21:57:15,583 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-metadata.json wandb-metadata.json |
| 2024-12-17 21:57:15,583 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log output.log |
| 2024-12-17 21:57:15,583 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/diff.patch diff.patch |
| 2024-12-17 21:57:15,589 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2024-12-17 21:57:15,589 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/2_eval_2_5329c53c34465295af0a.mp4 media/videos/eval/2_eval_2_5329c53c34465295af0a.mp4 |
| 2024-12-17 21:57:15,589 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/4_eval_4_08fea80871bf35a0fb30.mp4 media/videos/eval/4_eval_4_08fea80871bf35a0fb30.mp4 |
| 2024-12-17 21:57:15,589 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/1_eval_1_42cf14600b8d2071849c.mp4 media/videos/eval/1_eval_1_42cf14600b8d2071849c.mp4 |
| 2024-12-17 21:57:15,590 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/0_eval_0_dbd1eabcbea39a449528.mp4 media/videos/eval/0_eval_0_dbd1eabcbea39a449528.mp4 |
| 2024-12-17 21:57:15,590 INFO SenderThread:78001 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/media/videos/eval/3_eval_3_f7f27b273940874a00d0.mp4 media/videos/eval/3_eval_3_f7f27b273940874a00d0.mp4 |
| 2024-12-17 21:57:15,590 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 10 |
| 2024-12-17 21:57:15,590 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:57:15,590 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:15,590 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2024-12-17 21:57:15,590 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:15,590 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2024-12-17 21:57:15,590 INFO SenderThread:78001 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-17 21:57:15,773 INFO wandb-upload_0:78001 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/config.yaml |
| 2024-12-17 21:57:15,794 INFO wandb-upload_3:78001 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/requirements.txt |
| 2024-12-17 21:57:15,812 INFO wandb-upload_2:78001 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/output.log |
| 2024-12-17 21:57:15,813 INFO wandb-upload_1:78001 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/files/wandb-summary.json |
| 2024-12-17 21:57:16,013 INFO Thread-11 (_thread_body):78001 [sender.py:transition_state():614] send defer: 11 |
| 2024-12-17 21:57:16,014 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:16,014 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2024-12-17 21:57:16,014 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:16,014 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2024-12-17 21:57:16,014 INFO SenderThread:78001 [file_pusher.py:join():175] waiting for file pusher |
| 2024-12-17 21:57:16,017 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 12 |
| 2024-12-17 21:57:16,017 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:16,017 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2024-12-17 21:57:16,017 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:16,017 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2024-12-17 21:57:16,018 INFO SenderThread:78001 [file_stream.py:finish():601] file stream finish called |
| 2024-12-17 21:57:16,223 INFO SenderThread:78001 [file_stream.py:finish():605] file stream finish is done |
| 2024-12-17 21:57:16,223 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 13 |
| 2024-12-17 21:57:16,223 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:16,223 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2024-12-17 21:57:16,223 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:16,223 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2024-12-17 21:57:16,223 INFO SenderThread:78001 [sender.py:transition_state():614] send defer: 14 |
| 2024-12-17 21:57:16,223 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: defer |
| 2024-12-17 21:57:16,223 INFO HandlerThread:78001 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2024-12-17 21:57:16,224 DEBUG SenderThread:78001 [sender.py:send():379] send: final |
| 2024-12-17 21:57:16,224 DEBUG SenderThread:78001 [sender.py:send():379] send: footer |
| 2024-12-17 21:57:16,224 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: defer |
| 2024-12-17 21:57:16,224 INFO SenderThread:78001 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2024-12-17 21:57:16,224 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:57:16,224 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:57:16,224 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: server_info |
| 2024-12-17 21:57:16,225 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: poll_exit |
| 2024-12-17 21:57:16,225 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: server_info |
| 2024-12-17 21:57:16,226 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: get_summary |
| 2024-12-17 21:57:16,227 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: sampled_history |
| 2024-12-17 21:57:16,227 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: internal_messages |
| 2024-12-17 21:57:16,292 DEBUG SenderThread:78001 [sender.py:send_request():406] send_request: poll_exit |
| 2024-12-17 21:57:16,293 INFO MainThread:78001 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2024-12-17 21:57:16,293 INFO MainThread:78001 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2024-12-17 21:57:16,293 INFO MainThread:78001 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2024-12-17 21:57:16,293 DEBUG HandlerThread:78001 [handler.py:handle_request():158] handle_request: shutdown |
| 2024-12-17 21:57:16,293 INFO HandlerThread:78001 [handler.py:finish():882] shutting down handler |
| 2024-12-17 21:57:17,225 INFO WriterThread:78001 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/523_0/215240/wandb/run-20241217_215246-g6nsr00w/run-g6nsr00w.wandb |
| 2024-12-17 21:57:17,293 INFO SenderThread:78001 [sender.py:finish():1615] shutting down sender |
| 2024-12-17 21:57:17,293 INFO SenderThread:78001 [file_pusher.py:finish():169] shutting down file pusher |
| 2024-12-17 21:57:17,293 INFO SenderThread:78001 [file_pusher.py:join():175] waiting for file pusher |
|
|
