| 2026-07-10 15:07:34,147 INFO MainThread:18743 [wandb_setup.py:_flush():81] Current SDK version is 0.28.0 |
| 2026-07-10 15:07:34,147 INFO MainThread:18743 [wandb_setup.py:_flush():81] Configure stats pid to 18743 |
| 2026-07-10 15:07:34,147 INFO MainThread:18743 [wandb_setup.py:_flush():81] Loading settings from environment variables |
| 2026-07-10 15:07:34,151 INFO MainThread:18743 [wandb_init.py:setup_run_log_directory():725] Logging user logs to /mnt/kelp/minkyungkwon/seoul_world_mesh/ss3dm_train/shape_ft/wandb/run-20260710_150733-16p4iakm/logs/debug.log |
| 2026-07-10 15:07:34,152 INFO MainThread:18743 [wandb_init.py:setup_run_log_directory():726] Logging internal logs to /mnt/kelp/minkyungkwon/seoul_world_mesh/ss3dm_train/shape_ft/wandb/run-20260710_150733-16p4iakm/logs/debug-internal.log |
| 2026-07-10 15:07:34,152 INFO MainThread:18743 [wandb_init.py:init():768] calling init triggers |
| 2026-07-10 15:07:34,153 INFO MainThread:18743 [wandb_init.py:init():773] wandb.init called with sweep_config: {} |
| config: {'output_dir': '/mnt/kelp/minkyungkwon/seoul_world_mesh/ss3dm_train/shape_ft', '_wandb': {}} |
| 2026-07-10 15:07:34,153 INFO MainThread:18743 [wandb_init.py:init():816] starting backend |
| 2026-07-10 15:07:34,153 INFO MainThread:18743 [wandb_init.py:init():831] sending inform_init request |
| 2026-07-10 15:07:35,002 INFO MainThread:18743 [wandb_init.py:init():836] backend started and connected |
| 2026-07-10 15:07:35,004 INFO MainThread:18743 [wandb_init.py:init():906] updated telemetry |
| 2026-07-10 15:07:35,568 INFO MainThread:18743 [wandb_init.py:init():929] communicating run to backend with 90.0 second timeout |
| 2026-07-10 15:07:36,233 INFO MainThread:18743 [wandb_init.py:init():974] starting run threads in backend |
| 2026-07-10 15:07:36,363 INFO MainThread:18743 [wandb_run.py:_console_start():2523] atexit reg |
| 2026-07-10 15:07:36,364 INFO MainThread:18743 [wandb_run.py:_redirect():2373] redirect: wrap_raw |
| 2026-07-10 15:07:36,364 INFO MainThread:18743 [wandb_run.py:_redirect():2442] Wrapping output streams. |
| 2026-07-10 15:07:36,364 INFO MainThread:18743 [wandb_run.py:_redirect():2465] Redirects installed. |
| 2026-07-10 15:07:36,366 INFO MainThread:18743 [wandb_init.py:init():1012] run started, returning control to user process |
| 2026-07-12 15:32:24,499 INFO wandb-AsyncioManager-main:18743 [service_client.py:_forward_responses():122] [no run ID] Reached EOF. |
| 2026-07-12 15:32:24,500 INFO wandb-AsyncioManager-main:18743 [mailbox.py:close():154] [no run ID] Closing mailbox, abandoning 2 handles. |
| 2026-07-12 15:32:24,500 ERROR wandb-AsyncioManager-main:18743 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions |
| await fn() |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop |
| await asyncio_compat.race( |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race |
| async with open_task_group(race=True) as tg: |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/contextlib.py", line 206, in __aexit__ |
| await anext(self.gen) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group |
| await task_group._wait_all(race=race, timeout=exit_timeout) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all |
| raise exc |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all |
| result = await handle.wait_async(timeout=None) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async |
| response = await self._handle.wait_async(timeout=timeout) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async |
| raise HandleAbandonedError() |
| wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError |
| 2026-07-12 15:32:25,269 ERROR wandb-AsyncioManager-main:18743 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions |
| await fn() |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_stopping.py", line 87, in loop |
| should_stop = await self._check_should_stop() |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_stopping.py", line 115, in _check_should_stop |
| handle = await self._interface.deliver_async(request) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 58, in deliver_async |
| handle = await self._client.deliver(request) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 57, in deliver |
| handle = self._mailbox.require_response(request) |
| File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/mailbox/mailbox.py", line 89, in require_response |
| raise MailboxClosedError() |
| wandb.sdk.mailbox.mailbox.MailboxClosedError |
|
|
