| 2025-01-03 20:59:25,356 INFO StreamThr :1971983 [internal.py:wandb_internal():85] W&B internal server running at pid: 1971983, started at: 2025-01-03 20:59:25.355968 |
| 2025-01-03 20:59:25,357 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status |
| 2025-01-03 20:59:25,370 INFO WriterThread:1971983 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/run-75aq4eke.wandb |
| 2025-01-03 20:59:25,370 DEBUG SenderThread:1971983 [sender.py:send():379] send: header |
| 2025-01-03 20:59:25,378 DEBUG SenderThread:1971983 [sender.py:send():379] send: run |
| 2025-01-03 20:59:25,583 INFO SenderThread:1971983 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files |
| 2025-01-03 20:59:25,583 INFO SenderThread:1971983 [sender.py:_start_run_threads():1188] run started: 75aq4eke with start time 1735955965.356425 |
| 2025-01-03 20:59:25,599 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-03 20:59:25,599 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: check_version |
| 2025-01-03 20:59:25,645 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-03 20:59:25,650 DEBUG HandlerThread:1971983 [system_info.py:__init__():26] System info init |
| 2025-01-03 20:59:25,650 DEBUG HandlerThread:1971983 [system_info.py:__init__():41] System info init done |
| 2025-01-03 20:59:25,650 INFO HandlerThread:1971983 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-03 20:59:25,650 INFO SystemMonitor:1971983 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-03 20:59:25,650 INFO HandlerThread:1971983 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-03 20:59:25,650 INFO SystemMonitor:1971983 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-03 20:59:25,651 INFO SystemMonitor:1971983 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-03 20:59:25,652 INFO SystemMonitor:1971983 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-03 20:59:25,652 INFO SystemMonitor:1971983 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-03 20:59:25,653 INFO SystemMonitor:1971983 [interfaces.py:start():188] Started network monitoring |
| 2025-01-03 20:59:25,662 DEBUG HandlerThread:1971983 [system_info.py:probe():152] Probing system |
| 2025-01-03 20:59:25,664 DEBUG HandlerThread:1971983 [system_info.py:_probe_git():137] Probing git |
| 2025-01-03 20:59:25,669 DEBUG HandlerThread:1971983 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-03 20:59:25,669 DEBUG HandlerThread:1971983 [system_info.py:probe():200] Probing system done |
| 2025-01-03 20:59:25,669 DEBUG HandlerThread:1971983 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-04T01:59:25.662826', 'startedAt': '2025-01-04T01:59:25.347428', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'c537eb6734155592809c2a7db5cca971ec186d72'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3752.21675, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4502.095, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 4500.021, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 782.4653129577637}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-03 20:59:25,669 INFO HandlerThread:1971983 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-03 20:59:25,669 INFO HandlerThread:1971983 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-03 20:59:25,669 DEBUG HandlerThread:1971983 [system_info.py:_save_code():44] Saving code |
| 2025-01-03 20:59:25,670 DEBUG HandlerThread:1971983 [system_info.py:_save_code():67] Saving code done |
| 2025-01-03 20:59:25,670 DEBUG HandlerThread:1971983 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-03 20:59:25,732 DEBUG HandlerThread:1971983 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-03 20:59:25,733 INFO HandlerThread:1971983 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-03 20:59:25,736 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 20:59:25,736 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-03 20:59:25,736 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-03 20:59:25,736 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-03 20:59:25,969 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/motgmsf3-wandb-metadata.json |
| 2025-01-03 20:59:25,981 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-03 20:59:25,982 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-03 20:59:25,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:25,984 DEBUG SenderThread:1971983 [sender.py:send():379] send: telemetry |
| 2025-01-03 20:59:25,997 INFO wandb-upload_1:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/fl6n4962-code/FISH/eval_robot.py |
| 2025-01-03 20:59:26,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:59:26,025 INFO HandlerThread:1971983 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/tb" |
| save: true |
| } |
| control { |
| relay_id: "e539364ef70f43b69840a643d82a51e8" |
| } |
| _info { |
| stream_id: "75aq4eke" |
| } |
|
|
| 2025-01-03 20:59:26,025 DEBUG HandlerThread:1971983 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-03 20:59:26,026 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:59:26,041 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/vwybxkh7-diff.patch |
| 2025-01-03 20:59:26,165 DEBUG SenderThread:1971983 [sender.py:send():379] send: tbrecord |
| 2025-01-03 20:59:26,165 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 20:59:26,165 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 with policy live |
| 2025-01-03 20:59:26,383 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/rx0b9caa-events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 20:59:26,585 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 20:59:26,585 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/code/FISH/eval_robot.py |
| 2025-01-03 20:59:26,585 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:26,585 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/diff.patch |
| 2025-01-03 20:59:26,585 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-metadata.json |
| 2025-01-03 20:59:26,585 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/requirements.txt |
| 2025-01-03 20:59:26,586 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/code/FISH |
| 2025-01-03 20:59:26,586 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/code |
| 2025-01-03 20:59:26,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:27,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:28,584 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:28,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:29,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:30,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:59:30,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:31,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:32,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:33,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:34,649 DEBUG SenderThread:1971983 [sender.py:send():379] send: config |
| 2025-01-03 20:59:34,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:35,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:59:35,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:36,586 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:36,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:37,985 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:38,587 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:38,985 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:39,985 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:40,587 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:40,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:59:40,982 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:59:41,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:41,085 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:59:41,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:42,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:43,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:44,589 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:44,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:45,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:46,589 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:46,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:59:46,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:47,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:48,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:49,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:50,590 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 20:59:50,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:51,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:59:51,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:52,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:53,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:54,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:55,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 20:59:55,982 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 20:59:56,029 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:56,988 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:57,050 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 20:59:57,592 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/config.yaml |
| 2025-01-03 20:59:57,988 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:58,988 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 20:59:59,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:00,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:01,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:02,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:02,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:03,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:04,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:05,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:06,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:07,990 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:08,595 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:08,701 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:08,990 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:09,990 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:10,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:00:10,982 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:00:11,029 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:11,990 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:12,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:13,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:14,077 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:14,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:15,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:16,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:17,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:18,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:19,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:19,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:20,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:21,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:22,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:23,976 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:00:23,976 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_a512bbba28a2d223fb5f.mp4 with policy now |
| 2025-01-03 21:00:23,978 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:00:23,979 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:00:23,979 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:00:23,980 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:00:23,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:24,258 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/i3h0y8w7-media/videos/eval/0_eval_0_a512bbba28a2d223fb5f.mp4 |
| 2025-01-03 21:00:24,599 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:24,599 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/0_eval_0_a512bbba28a2d223fb5f.mp4 |
| 2025-01-03 21:00:24,599 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:00:24,600 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval |
| 2025-01-03 21:00:24,600 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos |
| 2025-01-03 21:00:24,600 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media |
| 2025-01-03 21:00:24,896 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/23d6nyag-media/videos/eval/0_eval_0_a512bbba28a2d223fb5f.mp4 |
| 2025-01-03 21:00:24,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:24,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:25,653 DEBUG SystemMonitor:1971983 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-03 21:00:25,654 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:00:25,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:00:25,982 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:00:26,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:26,600 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:26,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:27,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:28,600 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:28,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:29,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:30,455 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:30,601 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:30,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:31,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:32,602 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:32,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:33,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:34,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:35,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:35,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:36,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:37,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:38,994 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:39,994 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:40,604 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:00:40,877 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/zldt3mhn-events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:00:40,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:00:40,982 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:00:40,995 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:41,054 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:41,995 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:42,605 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:42,995 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:43,995 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:44,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:45,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:46,606 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:00:46,984 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:46,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:47,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:48,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:49,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:50,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:51,985 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:51,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:52,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:53,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:54,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:55,654 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:00:55,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:00:55,983 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:00:56,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:56,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:57,123 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:00:57,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:58,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:00:59,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:00,611 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:00,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:01,998 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:02,612 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:02,813 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:02,998 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:03,998 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:04,998 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:05,999 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:07,000 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:07,985 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:08,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:09,000 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:10,000 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:10,982 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:01:10,983 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:01:11,029 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:12,000 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:13,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:13,124 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:14,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:15,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:16,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:17,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:18,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:18,794 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:19,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:20,001 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:20,284 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:01:20,284 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_320233e3aec19848df18.mp4 with policy now |
| 2025-01-03 21:01:20,291 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:01:20,291 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:01:20,291 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:01:20,292 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:01:20,545 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/q58v7jky-media/videos/eval/1_eval_1_320233e3aec19848df18.mp4 |
| 2025-01-03 21:01:20,617 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:01:20,617 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:20,618 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/1_eval_1_320233e3aec19848df18.mp4 |
| 2025-01-03 21:01:20,618 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval |
| 2025-01-03 21:01:20,892 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/m2yex6v5-media/videos/eval/1_eval_1_320233e3aec19848df18.mp4 |
| 2025-01-03 21:01:21,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:22,010 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:22,618 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:23,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:23,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:24,010 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:25,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:25,654 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:01:25,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:01:25,987 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:01:26,029 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:26,620 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:27,005 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:28,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:28,621 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:29,005 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:29,802 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:30,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:31,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:31,521 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:01:31,521 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:01:31,522 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:01:31,622 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:01:31,622 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:01:32,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:32,623 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:33,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:34,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:35,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:35,691 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:36,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:36,625 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:37,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:38,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:39,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:40,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:40,701 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:40,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:01:40,983 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:01:41,007 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:42,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:42,626 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:43,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:44,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:44,627 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:01:45,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:45,988 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:46,008 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:47,009 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:48,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:49,010 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:50,010 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:50,989 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:51,010 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:52,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:53,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:54,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:55,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:55,655 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:01:55,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:01:55,983 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:01:56,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:56,083 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:01:57,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:58,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:01:59,011 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:00,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:01,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:01,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:02,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:03,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:04,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:05,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:06,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:07,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:07,991 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:08,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:09,012 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:10,017 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:10,559 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:02:10,559 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_1136994690265702739e.mp4 with policy now |
| 2025-01-03 21:02:10,563 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:02:10,563 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:02:10,564 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:02:10,566 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:02:10,636 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:02:10,636 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:10,636 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/2_eval_2_1136994690265702739e.mp4 |
| 2025-01-03 21:02:10,636 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval |
| 2025-01-03 21:02:10,829 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/0ow3ds6h-media/videos/eval/2_eval_2_1136994690265702739e.mp4 |
| 2025-01-03 21:02:10,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:02:10,983 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:02:11,017 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:11,091 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/uyvzrxb8-media/videos/eval/2_eval_2_1136994690265702739e.mp4 |
| 2025-01-03 21:02:12,016 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:12,636 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:13,017 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:13,127 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:14,021 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:15,020 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:16,022 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:16,638 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:17,018 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:18,019 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:18,639 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:18,690 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:19,018 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:20,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:21,020 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:22,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:23,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:23,994 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:24,026 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:25,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:25,655 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:02:25,985 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:02:25,986 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:02:26,029 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:27,026 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:28,022 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:29,022 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:29,296 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:02:29,296 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:02:29,297 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:29,298 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:02:29,642 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:02:29,643 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:02:29,889 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/jklp17ag-events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:02:30,027 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:30,643 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:31,027 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:32,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:33,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:34,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:34,644 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:34,737 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:35,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:36,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:37,024 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:38,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:39,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:39,994 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:40,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:40,983 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:02:40,983 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:02:41,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:42,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:43,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:44,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:45,025 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:45,064 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:46,026 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:47,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:48,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:49,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:50,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:50,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:51,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:52,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:53,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:54,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:55,030 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:55,454 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:02:55,454 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_036663fd181fb684915f.mp4 with policy now |
| 2025-01-03 21:02:55,457 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:02:55,457 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:02:55,457 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:02:55,458 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:02:55,651 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:02:55,651 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/3_eval_3_036663fd181fb684915f.mp4 |
| 2025-01-03 21:02:55,652 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval |
| 2025-01-03 21:02:55,656 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:02:55,750 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/kiil1o2x-media/videos/eval/3_eval_3_036663fd181fb684915f.mp4 |
| 2025-01-03 21:02:55,993 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:02:55,993 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:02:56,003 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/y207h7n6-media/videos/eval/3_eval_3_036663fd181fb684915f.mp4 |
| 2025-01-03 21:02:56,033 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:56,136 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:02:56,652 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:02:57,038 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:58,036 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:02:59,032 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:00,037 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:00,653 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:03:01,036 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:01,988 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:02,033 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:03,035 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:04,038 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:04,654 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:03:05,039 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:06,039 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:07,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:07,045 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:08,048 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:09,043 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:10,048 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:10,986 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:03:10,987 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:03:11,047 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:12,049 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:12,056 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:13,046 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:14,052 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:15,052 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:16,052 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:17,056 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:18,010 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:18,055 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:19,056 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:20,055 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:21,056 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:22,058 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:23,056 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:24,006 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:24,058 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:25,057 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:25,656 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:03:25,987 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:03:25,987 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:03:26,057 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:27,062 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:28,058 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:29,060 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:29,143 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:30,062 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:31,063 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:32,064 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:33,069 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:34,067 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:35,015 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:35,071 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:36,070 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:37,080 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:38,071 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:39,073 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:40,073 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:40,988 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:03:40,989 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:03:41,060 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:41,073 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:42,079 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:43,081 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:44,079 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:45,082 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:46,081 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:47,014 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:47,085 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:48,082 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:49,082 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:50,083 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:51,085 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:52,017 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:52,084 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:53,084 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:54,086 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:55,090 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:55,657 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:03:55,996 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:03:55,997 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:03:56,088 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:57,050 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:03:57,089 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:58,090 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:03:59,095 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:00,093 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:01,096 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:02,092 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:03,017 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:03,094 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:04,095 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:05,097 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:06,098 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:07,096 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:08,020 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:08,095 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:09,098 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:10,102 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:10,997 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:04:10,997 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:04:11,103 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:11,418 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:04:11,418 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:04:11,419 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:04:11,676 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:04:11,676 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:04:11,954 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/k9ly5vb7-events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:04:12,109 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:12,676 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:13,107 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:13,829 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:14,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:15,106 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:16,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:16,678 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:17,109 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:18,109 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:18,679 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:19,110 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:19,523 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:20,110 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:21,110 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:22,110 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:23,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:24,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:25,019 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:25,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:25,657 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:04:25,990 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:04:25,990 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:04:26,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:27,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:28,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:29,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:30,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:30,129 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:31,111 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:32,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:33,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:34,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:35,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:36,018 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:36,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:37,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:38,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:39,112 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:40,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:40,990 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:04:40,990 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:04:41,086 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:41,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:42,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:43,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:44,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:44,688 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:45,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:46,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:47,019 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:47,113 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:48,114 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:49,114 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:50,114 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:51,114 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:52,119 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:52,446 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:52,493 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:04:52,493 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:04:52,494 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_8a344ae7a6fd7d295455.mp4 with policy now |
| 2025-01-03 21:04:52,494 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:04:52,494 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:04:52,494 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:04:52,691 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:04:52,691 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:52,691 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/4_eval_4_8a344ae7a6fd7d295455.mp4 |
| 2025-01-03 21:04:52,692 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval |
| 2025-01-03 21:04:52,759 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/ipkj68j3-media/videos/eval/4_eval_4_8a344ae7a6fd7d295455.mp4 |
| 2025-01-03 21:04:53,014 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/z8r1oppo-media/videos/eval/4_eval_4_8a344ae7a6fd7d295455.mp4 |
| 2025-01-03 21:04:53,124 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:54,123 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:54,692 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:55,124 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:55,658 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:04:56,003 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:04:56,003 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:04:56,130 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:57,141 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:58,133 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:04:58,265 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:04:58,693 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:04:59,135 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:00,149 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:00,694 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:05:01,150 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:02,144 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:03,151 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:03,575 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:05:03,695 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:05:04,148 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:04,459 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-03 21:05:04,460 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-03 21:05:04,680 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:05:04,695 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:05:04,695 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json |
| 2025-01-03 21:05:04,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media |
| 2025-01-03 21:05:04,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table |
| 2025-01-03 21:05:04,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table/eval |
| 2025-01-03 21:05:04,908 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/otsy1mzd-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json |
| 2025-01-03 21:05:04,936 INFO wandb-upload_2:1971983 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpp9j1wpyh |
| 2025-01-03 21:05:05,019 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-03 21:05:05,028 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-03 21:05:05,145 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:05,325 INFO SenderThread:1971983 [sender.py:send_request_log_artifact():1525] logged artifact run-75aq4eke-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNzg4NzI2OA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTY3NjE2Mg==', 'latestArtifact': None}} |
| 2025-01-03 21:05:05,325 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:05:05,325 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,326 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,326 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:05:05,326 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json with policy now |
| 2025-01-03 21:05:05,326 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:05:05,327 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,330 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,330 DEBUG SenderThread:1971983 [sender.py:send():379] send: telemetry |
| 2025-01-03 21:05:05,330 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,331 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,331 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,331 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,331 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,331 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,332 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,332 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,332 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,332 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,333 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,333 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,333 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,333 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,334 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,334 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,334 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,334 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,335 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,335 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,335 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:05,335 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:05,335 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-03 21:05:05,563 INFO wandb-upload_1:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/hubmczio-media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json |
| 2025-01-03 21:05:05,672 INFO wandb-upload_2:1971983 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpjvfk30qd |
| 2025-01-03 21:05:05,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:05:05,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:05:05,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table/eval/total_success_rate_ci_6_6e34c8961e235c47938c.table.json |
| 2025-01-03 21:05:05,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table/eval |
| 2025-01-03 21:05:06,145 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:06,185 INFO SenderThread:1971983 [sender.py:send_request_log_artifact():1525] logged artifact run-75aq4eke-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQxNzg4NzI4MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjUyOTY3NjE2NA==', 'latestArtifact': None}} |
| 2025-01-03 21:05:06,185 DEBUG SenderThread:1971983 [sender.py:send():379] send: files |
| 2025-01-03 21:05:06,185 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_6_6e34c8961e235c47938c.table.json with policy now |
| 2025-01-03 21:05:06,185 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:05:06,185 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:06,186 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:06,357 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/eo5et8t7-media/table/eval/total_success_rate_ci_6_6e34c8961e235c47938c.table.json |
| 2025-01-03 21:05:06,620 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/3l8by8bd-media/table/eval/total_success_rate_ci_6_6e34c8961e235c47938c.table.json |
| 2025-01-03 21:05:06,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:05:06,696 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:05:07,145 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:08,145 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:09,145 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:09,186 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:05:10,145 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:10,992 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-03 21:05:10,993 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-03 21:05:11,146 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:12,146 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:13,146 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:14,146 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:15,082 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:05:15,146 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:15,666 DEBUG SenderThread:1971983 [sender.py:send():379] send: exit |
| 2025-01-03 21:05:15,666 INFO SenderThread:1971983 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-03 21:05:15,666 INFO SenderThread:1971983 [sender.py:send_exit():588] handling runtime: 350 |
| 2025-01-03 21:05:15,667 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:15,667 INFO SenderThread:1971983 [sender.py:send_exit():594] send defer |
| 2025-01-03 21:05:15,667 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:15,667 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-03 21:05:15,667 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:15,667 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-03 21:05:15,667 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-03 21:05:15,668 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:15,668 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-03 21:05:15,668 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:15,668 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-03 21:05:15,668 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-03 21:05:15,668 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:15,668 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-03 21:05:15,668 INFO HandlerThread:1971983 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-03 21:05:15,668 DEBUG SystemMonitor:1971983 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-03 21:05:15,668 DEBUG SystemMonitor:1971983 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-03 21:05:15,669 INFO HandlerThread:1971983 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-03 21:05:15,669 INFO HandlerThread:1971983 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-03 21:05:15,700 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:05:15,720 INFO HandlerThread:1971983 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-03 21:05:15,720 INFO HandlerThread:1971983 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-03 21:05:15,720 INFO HandlerThread:1971983 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-03 21:05:15,721 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:15,721 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-03 21:05:15,721 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-03 21:05:15,721 DEBUG SenderThread:1971983 [sender.py:send():379] send: stats |
| 2025-01-03 21:05:15,721 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:15,721 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-03 21:05:15,721 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:15,721 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-03 21:05:15,721 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-03 21:05:15,722 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:15,722 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-03 21:05:16,700 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:05:22,506 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 21:05:22,506 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:05:22,506 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-03 21:05:22,507 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:22,507 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-03 21:05:22,507 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-03 21:05:22,507 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 21:05:22,507 DEBUG SenderThread:1971983 [sender.py:send():379] send: history |
| 2025-01-03 21:05:22,508 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-03 21:05:22,508 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:22,508 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-03 21:05:22,509 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:22,509 DEBUG SenderThread:1971983 [sender.py:send():379] send: summary |
| 2025-01-03 21:05:22,513 INFO SenderThread:1971983 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-03 21:05:22,513 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:22,513 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-03 21:05:22,513 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-03 21:05:22,513 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:22,513 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-03 21:05:22,513 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:22,513 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-03 21:05:22,517 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-03 21:05:22,633 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-03 21:05:22,633 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:22,633 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-03 21:05:22,634 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:22,634 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-03 21:05:22,667 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 21:05:22,703 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/config.yaml |
| 2025-01-03 21:05:22,703 INFO Thread-12 :1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:05:26,383 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-03 21:05:26,383 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 21:05:26,384 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:26,384 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-03 21:05:26,384 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:26,384 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-03 21:05:26,384 INFO SenderThread:1971983 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-03 21:05:26,386 INFO SenderThread:1971983 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-03 21:05:26,389 INFO SenderThread:1971983 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-03 21:05:26,403 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-03 21:05:26,403 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:26,403 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-03 21:05:26,403 DEBUG SenderThread:1971983 [sender.py:send():379] send: artifact |
| 2025-01-03 21:05:26,609 INFO SenderThread:1971983 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQxNzg3MjQ4NQ==', 'state': 'COMMITTED', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQxNzg3MjQ4NQ==', 'versionIndex': 408}}} |
| 2025-01-03 21:05:26,609 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:26,609 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-03 21:05:26,609 INFO SenderThread:1971983 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-03 21:05:26,669 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 21:05:26,704 INFO SenderThread:1971983 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:05:26,704 INFO SenderThread:1971983 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files |
| 2025-01-03 21:05:26,704 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/config.yaml config.yaml |
| 2025-01-03 21:05:26,704 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:05:26,706 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json wandb-summary.json |
| 2025-01-03 21:05:26,706 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/requirements.txt requirements.txt |
| 2025-01-03 21:05:26,706 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-03 21:05:26,706 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log output.log |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/diff.patch diff.patch |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/3_eval_3_036663fd181fb684915f.mp4 media/videos/eval/3_eval_3_036663fd181fb684915f.mp4 |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/4_eval_4_8a344ae7a6fd7d295455.mp4 media/videos/eval/4_eval_4_8a344ae7a6fd7d295455.mp4 |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/2_eval_2_1136994690265702739e.mp4 media/videos/eval/2_eval_2_1136994690265702739e.mp4 |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/0_eval_0_a512bbba28a2d223fb5f.mp4 media/videos/eval/0_eval_0_a512bbba28a2d223fb5f.mp4 |
| 2025-01-03 21:05:26,707 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/videos/eval/1_eval_1_320233e3aec19848df18.mp4 media/videos/eval/1_eval_1_320233e3aec19848df18.mp4 |
| 2025-01-03 21:05:26,708 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json media/table/eval/success_rate_ci_5_bb2b66ada2f22e77fd7f.table.json |
| 2025-01-03 21:05:26,708 INFO SenderThread:1971983 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/media/table/eval/total_success_rate_ci_6_6e34c8961e235c47938c.table.json media/table/eval/total_success_rate_ci_6_6e34c8961e235c47938c.table.json |
| 2025-01-03 21:05:26,708 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-03 21:05:26,708 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 21:05:26,708 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:26,708 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-03 21:05:26,710 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:26,710 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-03 21:05:26,710 INFO SenderThread:1971983 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-03 21:05:26,910 INFO wandb-upload_0:1971983 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/config.yaml |
| 2025-01-03 21:05:26,943 INFO wandb-upload_2:1971983 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/requirements.txt |
| 2025-01-03 21:05:26,949 INFO wandb-upload_3:1971983 [upload_job.py:push():130] Uploaded file /tmp/tmpa3_yy03vwandb/613xjrxn-events.out.tfevents.1735955965.leonmkim-ROG-Strix-G15CS-G15CS.1971810.0 |
| 2025-01-03 21:05:26,957 INFO wandb-upload_4:1971983 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/wandb-summary.json |
| 2025-01-03 21:05:26,965 INFO wandb-upload_1:1971983 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/files/output.log |
| 2025-01-03 21:05:27,166 INFO Thread-11 (_thread_body):1971983 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-03 21:05:27,166 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:27,166 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-03 21:05:27,166 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:27,166 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-03 21:05:27,166 INFO SenderThread:1971983 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-03 21:05:27,167 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-03 21:05:27,168 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:27,168 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-03 21:05:27,168 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:27,168 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-03 21:05:27,168 INFO SenderThread:1971983 [file_stream.py:finish():601] file stream finish called |
| 2025-01-03 21:05:27,360 INFO SenderThread:1971983 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-03 21:05:27,360 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-03 21:05:27,360 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:27,360 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-03 21:05:27,360 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:27,360 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-03 21:05:27,360 INFO SenderThread:1971983 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-03 21:05:27,361 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-03 21:05:27,361 INFO HandlerThread:1971983 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-03 21:05:27,361 DEBUG SenderThread:1971983 [sender.py:send():379] send: final |
| 2025-01-03 21:05:27,361 DEBUG SenderThread:1971983 [sender.py:send():379] send: footer |
| 2025-01-03 21:05:27,361 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: defer |
| 2025-01-03 21:05:27,361 INFO SenderThread:1971983 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-03 21:05:27,361 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 21:05:27,362 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-03 21:05:27,362 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-03 21:05:27,362 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 21:05:27,362 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: server_info |
| 2025-01-03 21:05:27,364 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-03 21:05:27,364 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-03 21:05:27,365 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-03 21:05:27,425 DEBUG SenderThread:1971983 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-03 21:05:27,426 INFO MainThread:1971983 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-03 21:05:27,426 INFO MainThread:1971983 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-03 21:05:27,426 INFO MainThread:1971983 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-03 21:05:27,427 DEBUG HandlerThread:1971983 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-03 21:05:27,427 INFO HandlerThread:1971983 [handler.py:finish():882] shutting down handler |
| 2025-01-03 21:05:28,364 INFO WriterThread:1971983 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/1930_0/75aq4eke/wandb/run-20250103_205925-75aq4eke/run-75aq4eke.wandb |
| 2025-01-03 21:05:28,426 INFO SenderThread:1971983 [sender.py:finish():1615] shutting down sender |
| 2025-01-03 21:05:28,426 INFO SenderThread:1971983 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-03 21:05:28,426 INFO SenderThread:1971983 [file_pusher.py:join():175] waiting for file pusher |
|
|
