| 2025-01-18 19:54:57,688 INFO StreamThr :2566524 [internal.py:wandb_internal():85] W&B internal server running at pid: 2566524, started at: 2025-01-18 19:54:57.688322 |
| 2025-01-18 19:54:57,689 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status |
| 2025-01-18 19:54:57,701 INFO WriterThread:2566524 [datastore.py:open_for_write():87] open: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/run-mo0b6mh9.wandb |
| 2025-01-18 19:54:57,701 DEBUG SenderThread:2566524 [sender.py:send():379] send: header |
| 2025-01-18 19:54:57,710 DEBUG SenderThread:2566524 [sender.py:send():379] send: run |
| 2025-01-18 19:54:57,948 INFO SenderThread:2566524 [dir_watcher.py:__init__():211] watching files in: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files |
| 2025-01-18 19:54:57,949 INFO SenderThread:2566524 [sender.py:_start_run_threads():1188] run started: mo0b6mh9 with start time 1737248097.688871 |
| 2025-01-18 19:54:57,963 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: check_version |
| 2025-01-18 19:54:57,963 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: check_version |
| 2025-01-18 19:54:58,036 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: run_start |
| 2025-01-18 19:54:58,044 DEBUG HandlerThread:2566524 [system_info.py:__init__():26] System info init |
| 2025-01-18 19:54:58,044 DEBUG HandlerThread:2566524 [system_info.py:__init__():41] System info init done |
| 2025-01-18 19:54:58,045 INFO HandlerThread:2566524 [system_monitor.py:start():194] Starting system monitor |
| 2025-01-18 19:54:58,045 INFO SystemMonitor:2566524 [system_monitor.py:_start():158] Starting system asset monitoring threads |
| 2025-01-18 19:54:58,045 INFO HandlerThread:2566524 [system_monitor.py:probe():214] Collecting system info |
| 2025-01-18 19:54:58,045 INFO SystemMonitor:2566524 [interfaces.py:start():188] Started cpu monitoring |
| 2025-01-18 19:54:58,047 INFO SystemMonitor:2566524 [interfaces.py:start():188] Started disk monitoring |
| 2025-01-18 19:54:58,047 INFO SystemMonitor:2566524 [interfaces.py:start():188] Started gpu monitoring |
| 2025-01-18 19:54:58,048 INFO SystemMonitor:2566524 [interfaces.py:start():188] Started memory monitoring |
| 2025-01-18 19:54:58,048 INFO SystemMonitor:2566524 [interfaces.py:start():188] Started network monitoring |
| 2025-01-18 19:54:58,058 DEBUG HandlerThread:2566524 [system_info.py:probe():152] Probing system |
| 2025-01-18 19:54:58,060 DEBUG HandlerThread:2566524 [system_info.py:_probe_git():137] Probing git |
| 2025-01-18 19:54:58,064 DEBUG HandlerThread:2566524 [system_info.py:_probe_git():145] Probing git done |
| 2025-01-18 19:54:58,065 DEBUG HandlerThread:2566524 [system_info.py:probe():200] Probing system done |
| 2025-01-18 19:54:58,065 DEBUG HandlerThread:2566524 [system_monitor.py:probe():223] {'os': 'Linux-5.15.0-125-generic-x86_64-with-glibc2.31', 'python': '3.10.14', 'heartbeatAt': '2025-01-19T00:54:58.059031', 'startedAt': '2025-01-19T00:54:57.673346', 'docker': None, 'cuda': None, 'args': ('agent=diffusion', 'suite=frankagym', 'suite/frankagym_task@_global_=insertion'), 'state': 'running', 'program': '/home/leonmkim/fish_leon/FISH/eval_robot.py', 'codePathLocal': None, 'codePath': 'FISH/eval_robot.py', 'git': {'remote': 'https://github.com/leonmkim/fish_leon.git', 'commit': 'ec42957dc10d08dbd8b134558fcd8f9e470ccc31'}, 'email': 'leonmkim@seas.upenn.edu', 'root': '/home/leonmkim/fish_leon', 'host': 'leonmkim-ROG-Strix-G15CS-G15CS', 'username': 'leonmkim', 'executable': '/home/leonmkim/.pyenv/versions/lerobot/bin/python', 'cpu_count': 8, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3559.8128749999996, 'min': 800.0, 'max': 4700.0}, 'cpu_freq_per_core': [{'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 4499.999, 'min': 800.0, 'max': 4700.0}, {'current': 4489.241, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}, {'current': 3000.0, 'min': 800.0, 'max': 4700.0}], 'disk': {'/': {'total': 915.3232879638672, 'used': 579.0002593994141}}, 'gpu': 'NVIDIA GeForce RTX 2070 SUPER', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 2070 SUPER', 'memory_total': 8589934592}], 'memory': {'total': 62.71595764160156}} |
| 2025-01-18 19:54:58,065 INFO HandlerThread:2566524 [system_monitor.py:probe():224] Finished collecting system info |
| 2025-01-18 19:54:58,065 INFO HandlerThread:2566524 [system_monitor.py:probe():227] Publishing system info |
| 2025-01-18 19:54:58,065 DEBUG HandlerThread:2566524 [system_info.py:_save_code():44] Saving code |
| 2025-01-18 19:54:58,070 DEBUG HandlerThread:2566524 [system_info.py:_save_code():67] Saving code done |
| 2025-01-18 19:54:58,070 DEBUG HandlerThread:2566524 [system_info.py:_save_patches():84] Saving git patches |
| 2025-01-18 19:54:58,168 DEBUG HandlerThread:2566524 [system_info.py:_save_patches():126] Saving git patches done |
| 2025-01-18 19:54:58,169 INFO HandlerThread:2566524 [system_monitor.py:probe():229] Finished publishing system info |
| 2025-01-18 19:54:58,172 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:54:58,172 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now |
| 2025-01-18 19:54:58,172 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file code/FISH/eval_robot.py with policy now |
| 2025-01-18 19:54:58,173 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file diff.patch with policy now |
| 2025-01-18 19:54:58,173 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch with policy now |
| 2025-01-18 19:54:58,402 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: python_packages |
| 2025-01-18 19:54:58,403 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: python_packages |
| 2025-01-18 19:54:58,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:54:58,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:54:58,404 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:54:58,435 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/vx0a3pt4-diff.patch |
| 2025-01-18 19:54:58,438 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/ibb2j8co-wandb-metadata.json |
| 2025-01-18 19:54:58,445 INFO HandlerThread:2566524 [handler.py:handle_tbrecord():783] handling tbrecord: tbrecord { |
| log_dir: "/home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/tb" |
| save: true |
| } |
| control { |
| relay_id: "70c5eacc984a4afebe7199c0bf475f13" |
| } |
| _info { |
| stream_id: "mo0b6mh9" |
| } |
|
|
| 2025-01-18 19:54:58,445 DEBUG HandlerThread:2566524 [config_util.py:dict_from_config_file():70] no default config file found in config-defaults.yaml |
| 2025-01-18 19:54:58,452 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/x1yqpgel-upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-18 19:54:58,483 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/wt9sjipv-code/FISH/eval_robot.py |
| 2025-01-18 19:54:58,498 DEBUG SenderThread:2566524 [sender.py:send():379] send: telemetry |
| 2025-01-18 19:54:58,498 DEBUG SenderThread:2566524 [sender.py:send():379] send: tbrecord |
| 2025-01-18 19:54:58,499 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:54:58,499 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 with policy live |
| 2025-01-18 19:54:58,685 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/2ud0oelk-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:54:58,950 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:54:58,950 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/diff.patch |
| 2025-01-18 19:54:58,950 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-metadata.json |
| 2025-01-18 19:54:58,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/code/FISH/eval_robot.py |
| 2025-01-18 19:54:58,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/requirements.txt |
| 2025-01-18 19:54:58,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:54:58,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-18 19:54:58,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/code/FISH |
| 2025-01-18 19:54:58,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/code |
| 2025-01-18 19:54:59,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:00,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:00,949 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:01,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:02,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:02,836 DEBUG SenderThread:2566524 [sender.py:send():379] send: config |
| 2025-01-18 19:55:02,836 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:03,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:04,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:04,951 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:05,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:06,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:07,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:08,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:08,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:08,952 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:09,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:10,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:11,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:12,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:12,953 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:13,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:55:13,403 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:55:13,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:13,497 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:14,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:15,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:16,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:17,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:18,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:19,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:19,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:20,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:21,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:22,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:23,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:24,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:25,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:25,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:26,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:27,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:28,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:55:28,403 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:55:28,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:29,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:30,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:30,558 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:30,958 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/config.yaml |
| 2025-01-18 19:55:31,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:32,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:33,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:34,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:35,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:36,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:36,409 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:37,409 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:38,409 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:39,409 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:40,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:41,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:41,716 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:42,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:42,961 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:43,403 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:55:43,404 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:55:43,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:44,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:44,962 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:45,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:46,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:46,963 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:55:47,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:47,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:48,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:49,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:50,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:51,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:52,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:53,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:53,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:54,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:55,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:56,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:57,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:58,049 DEBUG SystemMonitor:2566524 [system_monitor.py:_start():172] Starting system metrics aggregation loop |
| 2025-01-18 19:55:58,049 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:55:58,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:55:58,404 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:55:58,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:55:58,463 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:55:59,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:00,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:01,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:02,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:03,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:04,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:04,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:05,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:06,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:07,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:08,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:09,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:09,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:10,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:11,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:12,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:13,362 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:56:13,363 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/0_eval_0_dd640c135955b3d54fec.mp4 with policy now |
| 2025-01-18 19:56:13,363 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 19:56:13,364 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:56:13,365 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:56:13,365 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:56:13,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:56:13,405 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:56:13,416 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:13,622 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/c33jqz39-media/videos/eval/0_eval_0_dd640c135955b3d54fec.mp4 |
| 2025-01-18 19:56:13,970 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/0_eval_0_dd640c135955b3d54fec.mp4 |
| 2025-01-18 19:56:13,970 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:56:13,970 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 19:56:13,970 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos |
| 2025-01-18 19:56:13,970 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media |
| 2025-01-18 19:56:14,242 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/5q0i49hy-media/videos/eval/0_eval_0_dd640c135955b3d54fec.mp4 |
| 2025-01-18 19:56:14,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:14,506 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:14,970 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:15,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:16,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:17,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:18,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:18,972 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:19,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:19,590 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:20,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:20,972 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:21,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:21,973 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:56:22,218 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/vqv8y13n-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:56:22,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:22,973 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:23,416 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:24,416 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:24,974 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:25,339 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:25,416 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:26,416 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:26,975 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:27,417 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:28,081 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:56:28,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:56:28,404 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:56:28,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:29,417 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:30,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:30,526 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:31,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:32,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:33,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:34,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:35,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:36,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:36,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:37,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:38,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:39,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:40,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:41,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:42,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:42,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:43,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:56:43,404 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:56:43,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:44,419 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:45,419 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:46,419 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:47,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:47,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:48,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:49,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:50,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:51,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:52,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:53,098 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:53,351 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:56:53,352 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/1_eval_1_f01db884f36bf3d225fe.mp4 with policy now |
| 2025-01-18 19:56:53,354 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 19:56:53,355 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:56:53,355 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:56:53,356 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:56:53,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:53,621 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/cj77boio-media/videos/eval/1_eval_1_f01db884f36bf3d225fe.mp4 |
| 2025-01-18 19:56:53,984 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:56:53,984 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/1_eval_1_f01db884f36bf3d225fe.mp4 |
| 2025-01-18 19:56:53,984 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 19:56:54,253 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/bkkpb6kx-media/videos/eval/1_eval_1_f01db884f36bf3d225fe.mp4 |
| 2025-01-18 19:56:54,423 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:54,984 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:55,425 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:56,423 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:57,440 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:58,050 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:56:58,409 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:56:58,409 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:56:58,453 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:56:58,565 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:56:58,985 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:56:59,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:00,431 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:00,986 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:01,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:02,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:03,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:03,634 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:03,860 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:57:03,860 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:57:03,861 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:57:03,987 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:57:03,987 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:57:04,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:04,987 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:05,431 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:06,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:06,988 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:07,438 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:08,432 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:08,989 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:09,409 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:09,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:10,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:11,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:12,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:13,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:57:13,405 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:57:13,449 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:14,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:14,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:15,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:16,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:17,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:18,434 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:19,434 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:20,410 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:20,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:21,434 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:22,434 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:23,434 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:24,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:25,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:26,315 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:26,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:27,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:28,051 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:57:28,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:57:28,406 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:57:28,453 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:28,996 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:29,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:30,436 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:31,436 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:31,538 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:32,436 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:33,436 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:34,436 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:35,439 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:36,051 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:57:36,051 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/2_eval_2_593caa1fa9ea606eafa5.mp4 with policy now |
| 2025-01-18 19:57:36,056 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 19:57:36,056 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:57:36,057 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:57:36,057 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:57:36,312 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/3nrtrf6b-media/videos/eval/2_eval_2_593caa1fa9ea606eafa5.mp4 |
| 2025-01-18 19:57:36,438 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:36,998 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:36,998 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:57:36,999 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/2_eval_2_593caa1fa9ea606eafa5.mp4 |
| 2025-01-18 19:57:36,999 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 19:57:37,069 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:37,284 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/1puhry0r-media/videos/eval/2_eval_2_593caa1fa9ea606eafa5.mp4 |
| 2025-01-18 19:57:37,440 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:38,439 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:38,999 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:39,451 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:40,440 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:41,000 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:41,448 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:42,446 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:42,483 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:43,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:57:43,408 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:57:43,446 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:44,457 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:45,001 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:45,451 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:46,448 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:47,447 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:48,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:48,450 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:49,456 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:50,470 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:50,701 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:57:50,701 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:57:50,701 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:57:51,003 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:51,003 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:57:51,004 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:57:51,303 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/p5shhvgr-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:57:51,452 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:52,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:53,004 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:53,421 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:53,464 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:54,454 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:55,004 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:57:55,454 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:56,454 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:57,454 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:58,051 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:57:58,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:57:58,406 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:57:58,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:57:58,475 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:57:59,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:00,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:01,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:02,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:03,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:04,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:04,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:05,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:06,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:07,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:08,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:09,412 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:09,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:10,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:11,458 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:12,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:13,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:58:13,406 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:58:13,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:14,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:14,945 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:15,011 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:15,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:16,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:17,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:18,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:19,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:20,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:20,459 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:21,485 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:22,320 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:58:22,320 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/3_eval_3_05651e72bfc1fe624e25.mp4 with policy now |
| 2025-01-18 19:58:22,323 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 19:58:22,323 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:58:22,323 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:58:22,324 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:58:22,461 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:22,571 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/on9jey56-media/videos/eval/3_eval_3_05651e72bfc1fe624e25.mp4 |
| 2025-01-18 19:58:23,014 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:23,014 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:58:23,014 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/3_eval_3_05651e72bfc1fe624e25.mp4 |
| 2025-01-18 19:58:23,014 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 19:58:23,303 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/ih02u9zo-media/videos/eval/3_eval_3_05651e72bfc1fe624e25.mp4 |
| 2025-01-18 19:58:23,467 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:24,479 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:25,015 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:25,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:25,482 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:26,482 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:27,015 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:27,486 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:28,052 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:58:28,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:58:28,415 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:58:28,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:29,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:30,502 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:30,881 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:31,017 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:31,498 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:32,017 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:58:32,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:32,751 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:58:32,751 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:58:32,751 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:58:33,017 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:33,018 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:58:33,494 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:34,494 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:35,018 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:35,500 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:35,921 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:36,496 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:37,019 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:37,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:38,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:39,020 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:58:39,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:40,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:41,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:41,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:42,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:43,413 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:58:43,414 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:58:43,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:44,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:45,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:46,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:46,517 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:47,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:48,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:49,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:50,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:51,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:52,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:52,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:53,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:54,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:55,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:56,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:57,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:58:57,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:58,052 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:58:58,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:58:58,415 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:58:58,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:58:59,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:00,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:01,492 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:02,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:02,492 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:03,492 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:04,504 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:04,665 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:59:04,665 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/4_eval_4_09e9e69382c41ac17586.mp4 with policy now |
| 2025-01-18 19:59:04,673 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 19:59:04,674 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:59:04,674 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:59:04,675 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:59:04,936 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/ibqypos3-media/videos/eval/4_eval_4_09e9e69382c41ac17586.mp4 |
| 2025-01-18 19:59:05,029 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:05,029 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:59:05,029 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/4_eval_4_09e9e69382c41ac17586.mp4 |
| 2025-01-18 19:59:05,029 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 19:59:05,299 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/b8yu4iz5-media/videos/eval/4_eval_4_09e9e69382c41ac17586.mp4 |
| 2025-01-18 19:59:05,499 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:06,508 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:07,030 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:07,511 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:07,706 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:08,508 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:09,504 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:10,507 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:11,031 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:11,503 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:12,508 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:13,032 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:13,289 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:13,438 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:59:13,438 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:59:13,525 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:14,522 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:15,509 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:16,520 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:17,524 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:18,441 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:18,524 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:19,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:20,517 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:21,521 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:22,532 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:22,810 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:59:22,810 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:59:22,810 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:59:23,035 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:59:23,035 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:59:23,290 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/rhopy0zc-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 19:59:23,532 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:23,868 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:24,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:25,036 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:25,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:26,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:27,532 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:28,052 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:59:28,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:59:28,414 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:59:28,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:29,037 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:29,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:29,567 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:30,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:31,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:32,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:33,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:34,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:35,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:35,533 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:36,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:37,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:38,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:39,041 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:39,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:40,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:40,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:41,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:42,534 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:43,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:59:43,414 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:59:43,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:44,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:45,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:45,550 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:46,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:47,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:48,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:49,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:50,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:51,418 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:51,536 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:52,536 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:53,536 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:54,541 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:55,047 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:55,106 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 19:59:55,106 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/5_eval_5_d80af6b216e8c15164e9.mp4 with policy now |
| 2025-01-18 19:59:55,118 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 19:59:55,119 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 19:59:55,119 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 19:59:55,119 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 19:59:55,380 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/4idcojoq-media/videos/eval/5_eval_5_d80af6b216e8c15164e9.mp4 |
| 2025-01-18 19:59:55,541 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:56,047 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 19:59:56,047 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/5_eval_5_d80af6b216e8c15164e9.mp4 |
| 2025-01-18 19:59:56,048 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 19:59:56,349 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/8k3jrrr9-media/videos/eval/5_eval_5_d80af6b216e8c15164e9.mp4 |
| 2025-01-18 19:59:56,538 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:57,048 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 19:59:57,175 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 19:59:57,548 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:58,053 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 19:59:58,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 19:59:58,415 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 19:59:58,540 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 19:59:59,549 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:00,556 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:01,049 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:01,569 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:02,556 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:02,761 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:03,050 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:03,559 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:04,558 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:04,861 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:00:04,861 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:00:04,861 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:00:05,051 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:00:05,051 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:00:05,560 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:06,576 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:07,051 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:07,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:07,925 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:08,567 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:09,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:10,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:11,053 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:11,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:12,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:13,239 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:13,414 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:00:13,414 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:00:13,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:14,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:15,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:16,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:17,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:18,429 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:18,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:19,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:20,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:21,056 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:21,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:22,562 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:23,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:23,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:24,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:25,058 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:25,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:26,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:27,059 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:27,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:28,053 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:00:28,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:00:28,415 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:00:28,532 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:28,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:29,060 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:29,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:30,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:31,563 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:32,564 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:33,061 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:33,564 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:34,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:34,564 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:35,564 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:36,581 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:37,063 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:37,570 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:37,641 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:00:37,641 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:00:37,642 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/6_eval_6_3602d914de1709b10240.mp4 with policy now |
| 2025-01-18 20:00:37,642 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:00:37,642 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:00:37,642 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:00:37,927 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/gpirp6zb-media/videos/eval/6_eval_6_3602d914de1709b10240.mp4 |
| 2025-01-18 20:00:38,063 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:00:38,063 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/6_eval_6_3602d914de1709b10240.mp4 |
| 2025-01-18 20:00:38,063 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:00:38,337 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/epmkbwxg-media/videos/eval/6_eval_6_3602d914de1709b10240.mp4 |
| 2025-01-18 20:00:38,581 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:39,064 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:39,439 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:39,573 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:40,583 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:41,586 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:42,591 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:43,065 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:43,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:00:43,430 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:00:43,581 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:44,575 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:45,066 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:45,353 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:45,591 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:46,590 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:47,585 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:48,590 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:48,920 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:00:48,920 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:00:48,921 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:00:49,068 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:49,068 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:00:49,068 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:00:49,321 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/fo54zghb-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:00:49,593 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:50,377 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:50,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:51,069 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:51,596 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:52,597 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:53,069 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:53,601 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:54,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:55,451 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:00:55,602 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:56,597 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:57,071 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:00:57,597 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:58,054 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:00:58,423 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:00:58,423 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:00:58,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:00:59,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:00,578 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:00,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:01,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:02,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:03,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:04,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:05,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:06,437 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:06,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:07,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:08,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:09,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:10,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:11,437 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:11,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:12,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:13,423 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:01:13,423 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:01:13,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:14,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:15,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:16,558 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:16,599 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:17,600 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:18,600 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:19,602 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:20,601 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:21,080 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:21,601 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:22,161 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:22,601 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:23,628 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:24,206 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:01:24,206 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/7_eval_7_960ad0adbe0863fa34f9.mp4 with policy now |
| 2025-01-18 20:01:24,210 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:01:24,210 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:01:24,210 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:01:24,210 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:01:24,461 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/845kbyic-media/videos/eval/7_eval_7_960ad0adbe0863fa34f9.mp4 |
| 2025-01-18 20:01:24,603 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:25,081 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:25,081 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:01:25,082 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/7_eval_7_960ad0adbe0863fa34f9.mp4 |
| 2025-01-18 20:01:25,082 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:01:25,403 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/x9ppir5n-media/videos/eval/7_eval_7_960ad0adbe0863fa34f9.mp4 |
| 2025-01-18 20:01:25,630 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:26,631 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:27,082 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:27,255 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:27,644 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:28,054 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:01:28,432 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:01:28,432 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:01:28,637 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:29,083 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:29,632 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:30,641 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:31,642 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:32,653 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:32,853 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:33,084 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:33,644 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:34,643 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:34,977 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:01:34,978 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:01:34,978 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:01:35,085 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:01:35,085 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:01:35,656 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:36,658 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:37,086 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:37,645 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:37,877 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:38,671 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:39,645 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:40,645 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:41,088 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:01:41,645 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:42,645 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:43,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:43,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:01:43,431 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:01:43,645 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:44,646 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:45,646 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:46,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:47,646 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:48,444 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:48,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:49,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:50,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:51,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:52,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:53,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:53,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:54,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:55,647 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:56,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:57,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:58,055 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:01:58,430 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:01:58,431 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:01:58,585 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:01:58,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:01:59,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:00,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:01,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:02,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:03,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:04,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:04,649 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:05,649 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:06,665 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:07,097 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:07,217 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:02:07,217 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/8_eval_8_4a18bdd252b8f5e209e5.mp4 with policy now |
| 2025-01-18 20:02:07,228 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:02:07,228 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:02:07,229 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:02:07,229 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:02:07,469 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/wttdunw1-media/videos/eval/8_eval_8_4a18bdd252b8f5e209e5.mp4 |
| 2025-01-18 20:02:07,662 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:08,098 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:02:08,098 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/8_eval_8_4a18bdd252b8f5e209e5.mp4 |
| 2025-01-18 20:02:08,098 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:02:08,370 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/4uqktciz-media/videos/eval/8_eval_8_4a18bdd252b8f5e209e5.mp4 |
| 2025-01-18 20:02:08,662 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:09,098 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:09,460 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:09,666 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:10,679 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:11,689 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:12,689 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:13,100 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:13,455 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:02:13,455 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:02:13,697 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:14,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:15,033 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:15,101 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:15,697 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:16,722 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:17,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:18,029 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:02:18,029 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:02:18,029 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:02:18,102 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:02:18,102 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:02:18,383 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/rjc30w1s-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:02:18,729 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:19,102 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:19,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:20,727 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:20,729 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:21,103 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:21,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:22,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:23,104 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:23,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:24,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:25,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:26,475 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:26,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:27,698 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:28,055 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:02:28,432 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:02:28,432 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:02:28,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:29,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:30,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:31,503 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:31,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:32,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:33,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:34,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:35,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:36,699 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:37,475 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:37,700 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:38,700 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:39,700 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:40,700 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:41,700 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:42,475 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:42,701 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:43,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:02:43,433 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:02:43,701 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:44,701 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:45,701 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:46,702 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:47,505 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:47,702 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:48,702 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:49,725 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:50,381 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:02:50,381 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/9_eval_9_eeb6222427bdaa35d11b.mp4 with policy now |
| 2025-01-18 20:02:50,415 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:02:50,415 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:02:50,415 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:02:50,415 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:02:50,671 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/ikkwxibk-media/videos/eval/9_eval_9_eeb6222427bdaa35d11b.mp4 |
| 2025-01-18 20:02:50,724 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:51,115 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:51,115 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:02:51,115 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/9_eval_9_eeb6222427bdaa35d11b.mp4 |
| 2025-01-18 20:02:51,115 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:02:51,402 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/jhlb5rn9-media/videos/eval/9_eval_9_eeb6222427bdaa35d11b.mp4 |
| 2025-01-18 20:02:51,725 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:52,726 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:53,116 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:53,474 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:53,730 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:54,729 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:55,116 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:55,740 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:56,729 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:57,735 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:58,056 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:02:58,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:02:58,445 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:02:58,579 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:02:58,754 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:02:59,118 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:02:59,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:00,753 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:01,082 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:03:01,082 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:03:01,083 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:03:01,119 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:03:01,119 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:03:01,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:02,756 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:03,119 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:03,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:04,499 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:04,754 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:05,774 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:06,764 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:07,121 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:07,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:08,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:09,510 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:09,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:10,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:11,123 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:11,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:12,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:13,433 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:03:13,433 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:03:13,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:14,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:14,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:15,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:16,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:17,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:18,749 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:19,749 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:20,494 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:20,749 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:21,749 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:22,749 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:23,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:24,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:25,494 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:25,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:26,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:27,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:28,056 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:03:28,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:03:28,435 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:03:28,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:29,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:30,506 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:30,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:31,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:32,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:33,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:34,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:35,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:36,494 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:36,751 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:37,776 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:38,111 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:03:38,111 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/10_eval_10_708e5d86baaf5c876d21.mp4 with policy now |
| 2025-01-18 20:03:38,119 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:03:38,119 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:03:38,119 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:03:38,120 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:03:38,133 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:03:38,134 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/10_eval_10_708e5d86baaf5c876d21.mp4 |
| 2025-01-18 20:03:38,134 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:03:38,379 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/silop977-media/videos/eval/10_eval_10_708e5d86baaf5c876d21.mp4 |
| 2025-01-18 20:03:38,642 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/soaimu9s-media/videos/eval/10_eval_10_708e5d86baaf5c876d21.mp4 |
| 2025-01-18 20:03:38,768 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:39,134 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:39,769 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:40,803 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:41,135 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:41,501 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:41,789 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:42,790 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:43,135 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:43,469 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:03:43,469 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:03:43,797 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:44,798 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:45,816 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:46,845 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:46,937 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:47,137 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:47,821 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:48,803 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:49,796 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:50,140 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:03:50,141 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:03:50,141 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:03:50,142 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:03:50,142 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:03:50,384 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/ncdu7a72-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:03:50,855 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:51,142 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:51,842 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:52,449 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:52,813 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:53,822 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:54,845 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:55,144 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:03:55,842 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:56,841 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:57,565 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:03:57,854 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:58,057 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:03:58,470 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:03:58,470 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:03:58,838 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:03:59,840 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:00,829 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:01,146 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:01,829 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:02,829 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:03,097 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:03,829 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:04,830 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:05,830 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:06,834 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:07,835 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:08,516 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:08,832 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:09,833 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:10,833 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:11,834 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:12,833 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:13,438 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:04:13,438 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:04:13,516 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:13,834 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:14,835 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:15,834 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:16,834 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:17,834 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:18,516 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:18,835 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:19,835 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:20,835 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:21,835 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:22,836 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:23,517 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:23,836 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:24,836 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:25,836 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:26,837 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:27,857 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:28,057 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:04:28,394 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:04:28,394 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/11_eval_11_5e48fceb469367c5942f.mp4 with policy now |
| 2025-01-18 20:04:28,417 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:04:28,417 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:04:28,417 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:04:28,418 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:04:28,470 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:04:28,470 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:04:28,538 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:28,661 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/447rwsgw-media/videos/eval/11_eval_11_5e48fceb469367c5942f.mp4 |
| 2025-01-18 20:04:28,859 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:29,157 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:29,157 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:04:29,157 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/11_eval_11_5e48fceb469367c5942f.mp4 |
| 2025-01-18 20:04:29,157 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:04:29,452 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/9vv93n32-media/videos/eval/11_eval_11_5e48fceb469367c5942f.mp4 |
| 2025-01-18 20:04:29,848 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:30,857 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:31,158 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:31,851 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:32,874 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:33,158 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:33,555 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:33,854 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:34,863 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:35,855 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:36,886 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:37,160 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:37,877 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:38,581 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:38,884 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:39,161 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:39,161 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:04:39,200 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:04:39,200 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:04:39,200 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:04:39,932 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:40,161 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:04:40,888 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:41,161 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:41,917 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:42,878 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:43,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:04:43,445 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:04:43,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:44,579 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:44,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:45,163 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:04:45,878 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:46,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:47,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:48,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:49,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:50,519 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:50,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:51,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:52,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:53,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:54,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:55,519 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:04:55,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:56,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:57,879 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:58,058 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:04:58,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:04:58,445 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:04:58,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:04:59,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:00,582 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:00,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:01,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:02,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:03,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:04,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:05,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:06,520 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:06,880 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:07,881 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:08,881 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:09,932 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:10,907 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:11,174 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:05:11,296 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:05:11,296 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/12_eval_12_7a56ddf9bcd287e59382.mp4 with policy now |
| 2025-01-18 20:05:11,301 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:05:11,301 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:05:11,302 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:05:11,302 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:05:11,554 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/os7081uj-media/videos/eval/12_eval_12_7a56ddf9bcd287e59382.mp4 |
| 2025-01-18 20:05:11,918 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:12,174 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:05:12,174 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/12_eval_12_7a56ddf9bcd287e59382.mp4 |
| 2025-01-18 20:05:12,174 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:05:12,385 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:12,458 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/86r0qhw1-media/videos/eval/12_eval_12_7a56ddf9bcd287e59382.mp4 |
| 2025-01-18 20:05:12,929 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:13,174 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:05:13,482 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:05:13,482 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:05:13,909 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:14,915 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:15,966 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:16,922 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:17,176 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:05:17,634 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:17,928 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:18,938 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:19,177 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:05:19,950 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:20,956 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:21,943 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:22,968 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:23,536 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:23,964 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:24,973 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:25,971 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:26,971 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:27,976 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:28,058 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:05:28,495 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:05:28,496 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:05:28,650 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:28,996 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:29,964 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:30,982 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:31,969 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:32,981 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:34,000 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:34,551 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:34,998 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:35,971 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:36,994 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:38,007 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:38,975 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:39,598 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:39,997 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:40,991 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:41,992 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:43,029 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:43,471 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:05:43,472 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:05:43,999 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:44,801 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:45,006 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:46,040 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:47,019 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:48,018 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:49,033 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:50,031 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:50,572 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:51,044 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:52,048 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:53,050 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:54,078 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:55,083 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:55,578 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:05:56,070 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:57,085 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:57,192 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:05:57,192 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:05:57,298 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:05:57,298 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:05:57,298 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:05:58,058 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:05:58,081 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:58,192 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:05:58,480 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:05:58,480 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:05:59,085 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:05:59,193 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:00,090 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:00,849 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:01,083 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:02,140 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:03,092 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:03,194 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:04,113 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:05,112 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:06,114 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:06,579 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:07,112 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:08,130 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:09,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:10,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:11,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:11,197 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:12,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:12,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:13,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:13,474 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:06:13,474 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:06:14,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:15,105 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:16,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:17,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:17,579 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:18,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:19,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:20,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:21,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:22,106 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:23,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:23,572 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:24,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:25,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:26,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:27,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:28,059 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:06:28,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:28,474 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:06:28,474 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:06:29,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:29,572 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:30,107 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:31,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:32,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:33,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:34,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:34,572 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:35,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:36,108 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:37,133 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:37,209 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:37,521 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:06:37,521 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/13_eval_13_ce205101458cc00950f1.mp4 with policy now |
| 2025-01-18 20:06:37,545 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:06:37,546 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:06:37,546 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:06:37,546 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:06:37,782 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/gm0yq7b0-media/videos/eval/13_eval_13_ce205101458cc00950f1.mp4 |
| 2025-01-18 20:06:38,124 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:38,210 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:06:38,210 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/13_eval_13_ce205101458cc00950f1.mp4 |
| 2025-01-18 20:06:38,210 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:06:38,499 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/26qywfdv-media/videos/eval/13_eval_13_ce205101458cc00950f1.mp4 |
| 2025-01-18 20:06:39,137 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:39,210 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:39,588 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:40,150 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:41,136 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:42,153 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:43,146 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:43,212 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:43,492 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:06:43,492 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:06:44,164 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:45,153 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:45,212 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:45,497 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:46,171 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:47,151 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:48,176 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:49,146 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:49,361 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:06:49,361 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:06:49,366 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:06:50,170 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:50,214 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:06:50,215 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:06:50,469 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/x6a11xcl-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:06:50,550 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:51,174 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:51,215 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:52,183 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:53,188 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:54,172 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:55,175 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:55,217 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:06:55,771 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:06:56,171 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:57,172 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:58,059 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:06:58,174 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:06:58,496 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:06:58,497 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:06:59,190 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:00,221 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:01,202 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:01,617 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:02,201 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:03,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:03,220 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:04,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:05,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:06,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:07,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:07,594 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:08,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:09,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:10,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:11,197 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:12,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:13,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:13,224 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:13,373 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:13,482 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:07:13,483 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:07:14,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:15,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:16,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:17,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:18,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:18,594 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:19,198 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:20,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:21,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:22,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:23,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:24,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:24,594 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:25,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:26,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:27,199 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:28,060 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:07:28,200 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:28,482 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:07:28,483 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:07:29,203 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:30,230 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:30,305 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:30,978 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:07:30,978 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/14_eval_14_bf5561b5a4f98495a054.mp4 with policy now |
| 2025-01-18 20:07:30,998 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:07:30,999 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:07:30,999 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:07:30,999 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:07:31,215 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:31,232 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:31,232 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:07:31,232 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/14_eval_14_bf5561b5a4f98495a054.mp4 |
| 2025-01-18 20:07:31,233 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:07:31,250 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/ojwd1tbi-media/videos/eval/14_eval_14_bf5561b5a4f98495a054.mp4 |
| 2025-01-18 20:07:31,517 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/fp2c2bo7-media/videos/eval/14_eval_14_bf5561b5a4f98495a054.mp4 |
| 2025-01-18 20:07:32,235 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:33,233 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:33,234 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:34,233 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:34,275 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:35,262 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:35,713 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:36,234 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:36,248 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:37,271 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:38,233 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:39,265 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:40,237 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:40,237 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:07:40,237 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:40,426 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:07:40,426 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:07:40,426 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:07:41,238 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:07:41,238 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:07:41,259 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:41,353 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:42,239 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:42,244 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:43,261 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:43,504 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:07:43,504 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:07:44,239 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:44,294 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:45,293 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:46,240 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:46,314 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:46,686 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:47,301 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:48,292 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:49,298 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:50,303 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:51,293 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:52,306 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:52,643 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:53,291 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:54,336 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:55,281 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:56,244 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:07:56,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:57,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:58,060 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:07:58,061 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:07:58,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:07:58,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:07:58,488 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:07:59,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:00,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:01,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:02,247 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:02,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:03,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:03,445 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:04,282 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:05,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:06,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:07,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:08,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:08,616 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:09,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:10,250 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:10,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:11,283 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:12,284 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:13,284 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:13,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:08:13,488 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:08:14,284 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:14,616 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:15,284 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:16,284 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:17,284 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:18,285 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:19,285 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:19,616 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:20,285 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:21,285 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:22,290 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:22,534 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:08:22,535 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/15_eval_15_2535ee93601a7918b8bf.mp4 with policy now |
| 2025-01-18 20:08:22,554 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:08:22,554 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:08:22,554 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:08:22,555 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:08:22,807 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/9oi6gffe-media/videos/eval/15_eval_15_2535ee93601a7918b8bf.mp4 |
| 2025-01-18 20:08:23,256 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:08:23,256 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/15_eval_15_2535ee93601a7918b8bf.mp4 |
| 2025-01-18 20:08:23,256 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:08:23,343 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:23,596 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/zjpz506b-media/videos/eval/15_eval_15_2535ee93601a7918b8bf.mp4 |
| 2025-01-18 20:08:24,256 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:24,301 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:24,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:25,257 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:25,339 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:26,325 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:27,329 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:28,061 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:08:28,258 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:28,316 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:28,556 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:08:28,556 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:08:29,355 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:29,669 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:30,354 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:31,357 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:32,260 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:32,383 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:32,487 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:08:32,488 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:08:32,488 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:08:33,260 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:08:33,260 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:08:33,362 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:34,261 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:34,348 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:34,689 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:35,390 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:36,261 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:36,357 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:37,393 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:38,262 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:38,377 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:39,377 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:40,263 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:40,387 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:40,644 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:41,389 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:42,392 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:43,419 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:43,540 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:08:43,540 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:08:44,404 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:45,428 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:45,678 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:46,431 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:47,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:48,267 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:48,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:49,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:50,267 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:08:50,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:51,405 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:51,623 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:52,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:53,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:54,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:55,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:56,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:56,623 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:08:57,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:58,061 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:08:58,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:08:58,501 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:08:58,501 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:08:59,406 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:00,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:01,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:02,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:02,623 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:03,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:04,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:05,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:06,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:07,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:07,623 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:08,407 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:09,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:10,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:11,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:12,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:13,086 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:13,408 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:13,501 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:09:13,501 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:09:14,278 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:14,435 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:15,420 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:15,798 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:09:15,798 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/16_eval_16_7e356bbb63bbb79e9934.mp4 with policy now |
| 2025-01-18 20:09:15,802 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:09:15,802 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:09:15,802 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:09:15,802 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:09:16,073 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/dxyaze9m-media/videos/eval/16_eval_16_7e356bbb63bbb79e9934.mp4 |
| 2025-01-18 20:09:16,279 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:16,279 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:09:16,279 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/16_eval_16_7e356bbb63bbb79e9934.mp4 |
| 2025-01-18 20:09:16,279 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:09:16,453 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:16,551 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/s8tnjemt-media/videos/eval/16_eval_16_7e356bbb63bbb79e9934.mp4 |
| 2025-01-18 20:09:17,279 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:17,454 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:18,280 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:18,478 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:18,663 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:19,443 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:20,452 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:21,477 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:22,282 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:22,462 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:23,465 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:23,665 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:24,282 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:24,450 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:25,475 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:26,471 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:27,284 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:09:27,452 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:27,541 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/53vvgyah-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:09:27,555 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:09:27,555 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:09:27,555 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:09:28,061 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:09:28,284 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:28,284 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:09:28,483 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:28,545 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:09:28,545 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:09:28,675 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:29,470 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:30,285 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:30,543 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:31,541 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:32,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:33,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:34,287 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:34,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:34,663 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:35,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:36,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:37,487 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:38,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:39,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:39,664 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:40,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:41,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:42,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:43,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:43,501 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:09:43,501 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:09:44,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:45,488 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:45,664 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:46,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:47,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:48,293 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:48,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:49,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:50,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:50,664 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:51,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:52,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:53,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:54,489 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:55,490 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:55,664 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:09:56,491 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:57,493 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:58,062 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:09:58,298 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:09:58,493 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:09:58,501 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:09:58,502 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:09:59,501 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:00,299 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:00,299 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/17_eval_17_21895a4eb18f20222f93.mp4 |
| 2025-01-18 20:10:00,299 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:10:00,338 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:10:00,338 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/17_eval_17_21895a4eb18f20222f93.mp4 with policy now |
| 2025-01-18 20:10:00,362 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:10:00,362 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:10:00,362 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:10:00,363 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:10:00,502 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:00,565 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/4vutdphb-media/videos/eval/17_eval_17_21895a4eb18f20222f93.mp4 |
| 2025-01-18 20:10:00,849 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/nmxvdykm-media/videos/eval/17_eval_17_21895a4eb18f20222f93.mp4 |
| 2025-01-18 20:10:01,299 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:10:01,476 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:01,524 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:02,300 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:02,539 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:03,300 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:03,561 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:04,586 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:05,551 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:06,302 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:06,551 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:07,181 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:07,578 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:08,602 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:09,601 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:09,606 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:10:09,606 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:10:09,607 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:10:10,303 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:10,304 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:10:10,304 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:10:10,611 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:11,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:12,304 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:12,629 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:12,661 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:13,535 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:10:13,536 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:10:13,640 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:14,607 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:15,607 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:16,306 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:16,607 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:17,607 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:18,209 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:18,608 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:19,608 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:20,608 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:21,608 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:22,309 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:22,608 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:23,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:23,680 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:24,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:25,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:26,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:27,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:28,062 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:10:28,506 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:10:28,507 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:10:28,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:28,680 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:29,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:30,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:31,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:32,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:33,609 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:33,680 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:34,610 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:35,610 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:36,610 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:37,610 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:38,610 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:38,680 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:39,610 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:40,611 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:41,654 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:42,318 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:42,613 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:10:42,613 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/18_eval_18_76a7bdc1f11678727d6f.mp4 with policy now |
| 2025-01-18 20:10:42,622 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:42,669 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:10:42,669 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:10:42,670 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:10:42,670 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:10:42,863 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/w2hj5yjy-media/videos/eval/18_eval_18_76a7bdc1f11678727d6f.mp4 |
| 2025-01-18 20:10:43,319 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:43,319 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:10:43,319 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/18_eval_18_76a7bdc1f11678727d6f.mp4 |
| 2025-01-18 20:10:43,319 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:10:43,541 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:10:43,541 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:10:43,604 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/hh67nec3-media/videos/eval/18_eval_18_76a7bdc1f11678727d6f.mp4 |
| 2025-01-18 20:10:43,622 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:43,722 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:44,319 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:44,659 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:45,320 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:45,662 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:46,648 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:47,642 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:48,321 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:48,653 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:48,738 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:49,674 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:50,703 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:51,688 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:52,323 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:52,685 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:53,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:54,720 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:10:54,750 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:55,735 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:56,664 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:10:56,664 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:10:56,664 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:10:56,714 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:57,325 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:10:57,325 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:10:57,748 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:58,063 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:10:58,325 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:10:58,542 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:10:58,542 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:10:58,715 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:10:59,719 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:00,326 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:00,719 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:00,719 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:01,706 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:02,328 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:02,706 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:03,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:04,329 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:04,706 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:05,706 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:06,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:06,710 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:07,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:08,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:09,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:10,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:11,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:11,710 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:12,707 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:13,508 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:11:13,509 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:11:13,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:14,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:15,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:16,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:16,710 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:17,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:18,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:19,708 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:20,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:21,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:21,711 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:22,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:23,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:24,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:25,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:26,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:26,711 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:27,709 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:28,063 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:11:28,546 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:11:28,546 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:11:28,729 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:29,340 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:11:29,340 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/videos/eval/19_eval_19_0dc41dd83150fd316589.mp4 with policy now |
| 2025-01-18 20:11:29,343 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/19_eval_19_0dc41dd83150fd316589.mp4 |
| 2025-01-18 20:11:29,343 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval |
| 2025-01-18 20:11:29,348 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:11:29,348 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:11:29,348 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:29,349 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:29,598 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/q9wwsyvm-media/videos/eval/19_eval_19_0dc41dd83150fd316589.mp4 |
| 2025-01-18 20:11:29,716 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:29,856 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/8dqa5vaf-media/videos/eval/19_eval_19_0dc41dd83150fd316589.mp4 |
| 2025-01-18 20:11:30,343 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:30,343 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:11:30,722 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:31,718 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:32,344 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:32,411 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:32,774 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:33,737 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:34,346 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:34,723 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:35,762 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:36,772 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:37,523 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:37,763 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:38,347 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:38,764 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:39,348 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:11:39,720 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:11:39,720 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:39,721 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:39,804 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:40,348 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:40,349 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:11:40,522 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-18 20:11:40,522 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-18 20:11:40,717 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:11:40,738 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:41,012 INFO wandb-upload_3:2566524 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpu0o677tu |
| 2025-01-18 20:11:41,074 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: log_artifact |
| 2025-01-18 20:11:41,085 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: partial_history |
| 2025-01-18 20:11:41,349 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:11:41,349 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/table/eval/success_rate_ci_20_6cabaaf3e79ec48751e3.table.json |
| 2025-01-18 20:11:41,350 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/table/eval/total_success_rate_ci_21_109ab103ab9a05e5d575.table.json |
| 2025-01-18 20:11:41,350 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media |
| 2025-01-18 20:11:41,350 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/table/eval |
| 2025-01-18 20:11:41,350 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_created():271] file/dir created: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/table |
| 2025-01-18 20:11:41,402 INFO SenderThread:2566524 [sender.py:send_request_log_artifact():1525] logged artifact run-mo0b6mh9-evalsuccess_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1MTk1Mzc3Mw==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MzA2Mzk2Nw==', 'latestArtifact': None}} |
| 2025-01-18 20:11:41,402 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:11:41,403 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/table/eval/success_rate_ci_20_6cabaaf3e79ec48751e3.table.json with policy now |
| 2025-01-18 20:11:41,403 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:11:41,403 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,403 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,404 DEBUG SenderThread:2566524 [sender.py:send():379] send: telemetry |
| 2025-01-18 20:11:41,404 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,407 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,408 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,408 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,408 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,409 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,409 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,409 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,409 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,410 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,410 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,410 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,410 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,411 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,411 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,411 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,412 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,412 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,412 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,412 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,413 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:41,413 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:41,413 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: log_artifact |
| 2025-01-18 20:11:41,623 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/yzdnup9g-media/table/eval/success_rate_ci_20_6cabaaf3e79ec48751e3.table.json |
| 2025-01-18 20:11:41,644 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/24g65024-media/table/eval/total_success_rate_ci_21_109ab103ab9a05e5d575.table.json |
| 2025-01-18 20:11:41,661 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/sr481jlx-events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:11:41,738 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:41,782 INFO wandb-upload_3:2566524 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpe6tbkd_4 |
| 2025-01-18 20:11:41,871 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/xi840sew-media/table/eval/success_rate_ci_20_6cabaaf3e79ec48751e3.table.json |
| 2025-01-18 20:11:42,179 INFO SenderThread:2566524 [sender.py:send_request_log_artifact():1525] logged artifact run-mo0b6mh9-evaltotal_success_rate_ci - {'id': 'QXJ0aWZhY3Q6MTQ1MTk1Mzc4MQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjU0MzA2Mzk2OA==', 'latestArtifact': None}} |
| 2025-01-18 20:11:42,179 DEBUG SenderThread:2566524 [sender.py:send():379] send: files |
| 2025-01-18 20:11:42,179 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file media/table/eval/total_success_rate_ci_21_109ab103ab9a05e5d575.table.json with policy now |
| 2025-01-18 20:11:42,180 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:11:42,180 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:11:42,180 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:11:42,350 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:11:42,428 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /tmp/tmpwgaaf6o2wandb/44lpp5dk-media/table/eval/total_success_rate_ci_21_109ab103ab9a05e5d575.table.json |
| 2025-01-18 20:11:42,738 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:43,185 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:43,350 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/config.yaml |
| 2025-01-18 20:11:43,513 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:11:43,513 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:11:43,738 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:44,351 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:11:44,738 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:45,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:46,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:47,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:48,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:48,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:49,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:50,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:51,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:52,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:53,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:53,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:54,740 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:55,740 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:56,740 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:57,740 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:58,064 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:11:58,513 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:11:58,513 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:11:58,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:11:58,741 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:11:59,741 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:00,741 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:01,741 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:02,741 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:03,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:04,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:12:04,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:05,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:06,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:07,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:08,743 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:09,739 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:12:09,743 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:10,743 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:11,743 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:12,743 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:13,513 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: stop_status |
| 2025-01-18 20:12:13,513 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: stop_status |
| 2025-01-18 20:12:13,743 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:14,021 DEBUG SenderThread:2566524 [sender.py:send():379] send: exit |
| 2025-01-18 20:12:14,021 INFO SenderThread:2566524 [sender.py:send_exit():586] handling exit code: 0 |
| 2025-01-18 20:12:14,021 INFO SenderThread:2566524 [sender.py:send_exit():588] handling runtime: 1035 |
| 2025-01-18 20:12:14,022 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:12:14,022 INFO SenderThread:2566524 [sender.py:send_exit():594] send defer |
| 2025-01-18 20:12:14,022 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:14,022 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 0 |
| 2025-01-18 20:12:14,022 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:14,022 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 0 |
| 2025-01-18 20:12:14,022 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 1 |
| 2025-01-18 20:12:14,022 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:14,023 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 1 |
| 2025-01-18 20:12:14,023 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:14,023 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 1 |
| 2025-01-18 20:12:14,023 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 2 |
| 2025-01-18 20:12:14,023 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:14,023 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 2 |
| 2025-01-18 20:12:14,023 INFO HandlerThread:2566524 [system_monitor.py:finish():203] Stopping system monitor |
| 2025-01-18 20:12:14,023 DEBUG SystemMonitor:2566524 [system_monitor.py:_start():179] Finished system metrics aggregation loop |
| 2025-01-18 20:12:14,023 DEBUG SystemMonitor:2566524 [system_monitor.py:_start():183] Publishing last batch of metrics |
| 2025-01-18 20:12:14,023 INFO HandlerThread:2566524 [interfaces.py:finish():200] Joined cpu monitor |
| 2025-01-18 20:12:14,024 INFO HandlerThread:2566524 [interfaces.py:finish():200] Joined disk monitor |
| 2025-01-18 20:12:14,074 INFO HandlerThread:2566524 [interfaces.py:finish():200] Joined gpu monitor |
| 2025-01-18 20:12:14,074 INFO HandlerThread:2566524 [interfaces.py:finish():200] Joined memory monitor |
| 2025-01-18 20:12:14,074 INFO HandlerThread:2566524 [interfaces.py:finish():200] Joined network monitor |
| 2025-01-18 20:12:14,074 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:14,074 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 2 |
| 2025-01-18 20:12:14,074 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 3 |
| 2025-01-18 20:12:14,075 DEBUG SenderThread:2566524 [sender.py:send():379] send: stats |
| 2025-01-18 20:12:14,075 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:14,075 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 3 |
| 2025-01-18 20:12:14,075 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:14,075 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 3 |
| 2025-01-18 20:12:14,075 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 4 |
| 2025-01-18 20:12:14,075 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:14,075 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 4 |
| 2025-01-18 20:12:14,366 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:12:16,367 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:12:20,730 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-18 20:12:20,730 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:12:20,730 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: keepalive |
| 2025-01-18 20:12:20,730 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:12:20,731 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:20,731 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 4 |
| 2025-01-18 20:12:20,731 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 5 |
| 2025-01-18 20:12:20,731 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-18 20:12:20,732 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:20,732 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 5 |
| 2025-01-18 20:12:20,740 DEBUG SenderThread:2566524 [sender.py:send():379] send: history |
| 2025-01-18 20:12:20,740 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: summary_record |
| 2025-01-18 20:12:20,740 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:12:20,741 DEBUG SenderThread:2566524 [sender.py:send():379] send: summary |
| 2025-01-18 20:12:20,741 INFO SenderThread:2566524 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end |
| 2025-01-18 20:12:20,741 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:20,741 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 5 |
| 2025-01-18 20:12:20,741 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 6 |
| 2025-01-18 20:12:20,741 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:20,741 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 6 |
| 2025-01-18 20:12:20,742 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:20,742 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 6 |
| 2025-01-18 20:12:20,742 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 7 |
| 2025-01-18 20:12:20,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: status_report |
| 2025-01-18 20:12:20,742 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:20,742 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 7 |
| 2025-01-18 20:12:20,742 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:20,742 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 7 |
| 2025-01-18 20:12:21,022 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-18 20:12:21,370 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:12:22,841 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 8 |
| 2025-01-18 20:12:22,841 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-18 20:12:22,842 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:22,842 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 8 |
| 2025-01-18 20:12:22,842 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:22,842 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 8 |
| 2025-01-18 20:12:22,842 INFO SenderThread:2566524 [job_builder.py:build():440] Attempting to build job artifact |
| 2025-01-18 20:12:22,845 INFO SenderThread:2566524 [job_builder.py:_get_source_type():569] is repo sourced job |
| 2025-01-18 20:12:22,846 INFO SenderThread:2566524 [job_builder.py:build():545] adding wandb-job metadata file |
| 2025-01-18 20:12:22,861 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 9 |
| 2025-01-18 20:12:22,861 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:22,861 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 9 |
| 2025-01-18 20:12:22,862 DEBUG SenderThread:2566524 [sender.py:send():379] send: artifact |
| 2025-01-18 20:12:23,023 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-18 20:12:23,239 INFO wandb-upload_2:2566524 [upload_job.py:push():85] Skipped uploading /home/leonmkim/.local/share/wandb/artifacts/staging/tmpe9xdfu6j |
| 2025-01-18 20:12:23,239 INFO wandb-upload_0:2566524 [upload_job.py:push():85] Skipped uploading /tmp/tmp6zpygqk6/wandb-job.json |
| 2025-01-18 20:12:23,359 INFO wandb-upload_3:2566524 [upload_job.py:push():88] Uploaded file /home/leonmkim/.local/share/wandb/artifacts/staging/tmpi1brqpjv |
| 2025-01-18 20:12:23,371 INFO Thread-12 :2566524 [dir_watcher.py:_on_file_modified():288] file/dir modified: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:12:23,938 INFO SenderThread:2566524 [sender.py:send_artifact():1537] sent artifact job-https___github.com_leonmkim_fish_leon.git_FISH_eval_robot.py - {'id': 'QXJ0aWZhY3Q6MTQ1MTk1NDMyOQ==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjMxNzk5NjkyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1MTkxOTkxNA==', 'versionIndex': 448}}} |
| 2025-01-18 20:12:23,938 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:23,938 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 9 |
| 2025-01-18 20:12:23,938 INFO SenderThread:2566524 [dir_watcher.py:finish():358] shutting down directory watcher |
| 2025-01-18 20:12:24,373 INFO SenderThread:2566524 [dir_watcher.py:finish():388] scan: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files |
| 2025-01-18 20:12:24,373 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/config.yaml config.yaml |
| 2025-01-18 20:12:24,373 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 events.out.tfevents.1737248098.leonmkim-ROG-Strix-G15CS-G15CS.2566358.0 |
| 2025-01-18 20:12:24,373 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json wandb-summary.json |
| 2025-01-18 20:12:24,375 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/requirements.txt requirements.txt |
| 2025-01-18 20:12:24,375 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-metadata.json wandb-metadata.json |
| 2025-01-18 20:12:24,375 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch upstream_diff_0f018502ce3c0d2ad4dfb21f3d7778b9e3db3a78.patch |
| 2025-01-18 20:12:24,375 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log output.log |
| 2025-01-18 20:12:24,375 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/diff.patch diff.patch |
| 2025-01-18 20:12:24,375 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/code/FISH/eval_robot.py code/FISH/eval_robot.py |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/5_eval_5_d80af6b216e8c15164e9.mp4 media/videos/eval/5_eval_5_d80af6b216e8c15164e9.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/2_eval_2_593caa1fa9ea606eafa5.mp4 media/videos/eval/2_eval_2_593caa1fa9ea606eafa5.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/14_eval_14_bf5561b5a4f98495a054.mp4 media/videos/eval/14_eval_14_bf5561b5a4f98495a054.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/6_eval_6_3602d914de1709b10240.mp4 media/videos/eval/6_eval_6_3602d914de1709b10240.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/16_eval_16_7e356bbb63bbb79e9934.mp4 media/videos/eval/16_eval_16_7e356bbb63bbb79e9934.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/17_eval_17_21895a4eb18f20222f93.mp4 media/videos/eval/17_eval_17_21895a4eb18f20222f93.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/0_eval_0_dd640c135955b3d54fec.mp4 media/videos/eval/0_eval_0_dd640c135955b3d54fec.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/10_eval_10_708e5d86baaf5c876d21.mp4 media/videos/eval/10_eval_10_708e5d86baaf5c876d21.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/8_eval_8_4a18bdd252b8f5e209e5.mp4 media/videos/eval/8_eval_8_4a18bdd252b8f5e209e5.mp4 |
| 2025-01-18 20:12:24,380 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/4_eval_4_09e9e69382c41ac17586.mp4 media/videos/eval/4_eval_4_09e9e69382c41ac17586.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/13_eval_13_ce205101458cc00950f1.mp4 media/videos/eval/13_eval_13_ce205101458cc00950f1.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/12_eval_12_7a56ddf9bcd287e59382.mp4 media/videos/eval/12_eval_12_7a56ddf9bcd287e59382.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/15_eval_15_2535ee93601a7918b8bf.mp4 media/videos/eval/15_eval_15_2535ee93601a7918b8bf.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/11_eval_11_5e48fceb469367c5942f.mp4 media/videos/eval/11_eval_11_5e48fceb469367c5942f.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/1_eval_1_f01db884f36bf3d225fe.mp4 media/videos/eval/1_eval_1_f01db884f36bf3d225fe.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/7_eval_7_960ad0adbe0863fa34f9.mp4 media/videos/eval/7_eval_7_960ad0adbe0863fa34f9.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/19_eval_19_0dc41dd83150fd316589.mp4 media/videos/eval/19_eval_19_0dc41dd83150fd316589.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/3_eval_3_05651e72bfc1fe624e25.mp4 media/videos/eval/3_eval_3_05651e72bfc1fe624e25.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/9_eval_9_eeb6222427bdaa35d11b.mp4 media/videos/eval/9_eval_9_eeb6222427bdaa35d11b.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/videos/eval/18_eval_18_76a7bdc1f11678727d6f.mp4 media/videos/eval/18_eval_18_76a7bdc1f11678727d6f.mp4 |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/table/eval/success_rate_ci_20_6cabaaf3e79ec48751e3.table.json media/table/eval/success_rate_ci_20_6cabaaf3e79ec48751e3.table.json |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [dir_watcher.py:finish():402] scan save: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/media/table/eval/total_success_rate_ci_21_109ab103ab9a05e5d575.table.json media/table/eval/total_success_rate_ci_21_109ab103ab9a05e5d575.table.json |
| 2025-01-18 20:12:24,381 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 10 |
| 2025-01-18 20:12:24,382 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-18 20:12:24,382 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:24,382 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 10 |
| 2025-01-18 20:12:24,382 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:24,382 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 10 |
| 2025-01-18 20:12:24,382 INFO SenderThread:2566524 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-18 20:12:24,566 INFO wandb-upload_1:2566524 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/config.yaml |
| 2025-01-18 20:12:24,581 INFO wandb-upload_3:2566524 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/output.log |
| 2025-01-18 20:12:24,583 INFO wandb-upload_2:2566524 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/requirements.txt |
| 2025-01-18 20:12:24,589 INFO wandb-upload_0:2566524 [upload_job.py:push():130] Uploaded file /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/files/wandb-summary.json |
| 2025-01-18 20:12:24,789 INFO Thread-11 (_thread_body):2566524 [sender.py:transition_state():614] send defer: 11 |
| 2025-01-18 20:12:24,789 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:24,789 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 11 |
| 2025-01-18 20:12:24,790 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:24,790 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 11 |
| 2025-01-18 20:12:24,790 INFO SenderThread:2566524 [file_pusher.py:join():175] waiting for file pusher |
| 2025-01-18 20:12:24,793 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 12 |
| 2025-01-18 20:12:24,793 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:24,793 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 12 |
| 2025-01-18 20:12:24,793 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:24,793 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 12 |
| 2025-01-18 20:12:24,793 INFO SenderThread:2566524 [file_stream.py:finish():601] file stream finish called |
| 2025-01-18 20:12:25,014 INFO SenderThread:2566524 [file_stream.py:finish():605] file stream finish is done |
| 2025-01-18 20:12:25,014 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 13 |
| 2025-01-18 20:12:25,014 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:25,014 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 13 |
| 2025-01-18 20:12:25,015 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:25,015 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 13 |
| 2025-01-18 20:12:25,015 INFO SenderThread:2566524 [sender.py:transition_state():614] send defer: 14 |
| 2025-01-18 20:12:25,015 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: defer |
| 2025-01-18 20:12:25,015 INFO HandlerThread:2566524 [handler.py:handle_request_defer():184] handle defer: 14 |
| 2025-01-18 20:12:25,015 DEBUG SenderThread:2566524 [sender.py:send():379] send: final |
| 2025-01-18 20:12:25,015 DEBUG SenderThread:2566524 [sender.py:send():379] send: footer |
| 2025-01-18 20:12:25,015 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: defer |
| 2025-01-18 20:12:25,015 INFO SenderThread:2566524 [sender.py:send_request_defer():610] handle sender defer: 14 |
| 2025-01-18 20:12:25,016 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-18 20:12:25,016 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: server_info |
| 2025-01-18 20:12:25,016 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-18 20:12:25,016 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: poll_exit |
| 2025-01-18 20:12:25,016 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: server_info |
| 2025-01-18 20:12:25,017 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: get_summary |
| 2025-01-18 20:12:25,018 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: sampled_history |
| 2025-01-18 20:12:25,019 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: internal_messages |
| 2025-01-18 20:12:25,078 DEBUG SenderThread:2566524 [sender.py:send_request():406] send_request: poll_exit |
| 2025-01-18 20:12:25,078 INFO MainThread:2566524 [wandb_run.py:_footer_history_summary_info():4016] rendering history |
| 2025-01-18 20:12:25,079 INFO MainThread:2566524 [wandb_run.py:_footer_history_summary_info():4048] rendering summary |
| 2025-01-18 20:12:25,079 INFO MainThread:2566524 [wandb_run.py:_footer_sync_info():3975] logging synced files |
| 2025-01-18 20:12:25,080 DEBUG HandlerThread:2566524 [handler.py:handle_request():158] handle_request: shutdown |
| 2025-01-18 20:12:25,080 INFO HandlerThread:2566524 [handler.py:finish():882] shutting down handler |
| 2025-01-18 20:12:26,017 INFO WriterThread:2566524 [datastore.py:close():296] close: /home/leonmkim/fish_leon/FISH/exp_local/frankagym_pixels/FrankaInsertion-v1/3465_0/mo0b6mh9/wandb/run-20250118_195457-mo0b6mh9/run-mo0b6mh9.wandb |
| 2025-01-18 20:12:26,078 INFO SenderThread:2566524 [sender.py:finish():1615] shutting down sender |
| 2025-01-18 20:12:26,079 INFO SenderThread:2566524 [file_pusher.py:finish():169] shutting down file pusher |
| 2025-01-18 20:12:26,079 INFO SenderThread:2566524 [file_pusher.py:join():175] waiting for file pusher |
|
|
