shape_ft/wandb/debug.log

2026-07-10 15:07:34,147 INFO    MainThread:18743 [wandb_setup.py:_flush():81] Current SDK version is 0.28.0
2026-07-10 15:07:34,147 INFO    MainThread:18743 [wandb_setup.py:_flush():81] Configure stats pid to 18743
2026-07-10 15:07:34,147 INFO    MainThread:18743 [wandb_setup.py:_flush():81] Loading settings from environment variables
2026-07-10 15:07:34,151 INFO    MainThread:18743 [wandb_init.py:setup_run_log_directory():725] Logging user logs to /mnt/kelp/minkyungkwon/seoul_world_mesh/ss3dm_train/shape_ft/wandb/run-20260710_150733-16p4iakm/logs/debug.log
2026-07-10 15:07:34,152 INFO    MainThread:18743 [wandb_init.py:setup_run_log_directory():726] Logging internal logs to /mnt/kelp/minkyungkwon/seoul_world_mesh/ss3dm_train/shape_ft/wandb/run-20260710_150733-16p4iakm/logs/debug-internal.log
2026-07-10 15:07:34,152 INFO    MainThread:18743 [wandb_init.py:init():768] calling init triggers
2026-07-10 15:07:34,153 INFO    MainThread:18743 [wandb_init.py:init():773] wandb.init called with sweep_config: {}
config: {'output_dir': '/mnt/kelp/minkyungkwon/seoul_world_mesh/ss3dm_train/shape_ft', '_wandb': {}}
2026-07-10 15:07:34,153 INFO    MainThread:18743 [wandb_init.py:init():816] starting backend
2026-07-10 15:07:34,153 INFO    MainThread:18743 [wandb_init.py:init():831] sending inform_init request
2026-07-10 15:07:35,002 INFO    MainThread:18743 [wandb_init.py:init():836] backend started and connected
2026-07-10 15:07:35,004 INFO    MainThread:18743 [wandb_init.py:init():906] updated telemetry
2026-07-10 15:07:35,568 INFO    MainThread:18743 [wandb_init.py:init():929] communicating run to backend with 90.0 second timeout
2026-07-10 15:07:36,233 INFO    MainThread:18743 [wandb_init.py:init():974] starting run threads in backend
2026-07-10 15:07:36,363 INFO    MainThread:18743 [wandb_run.py:_console_start():2523] atexit reg
2026-07-10 15:07:36,364 INFO    MainThread:18743 [wandb_run.py:_redirect():2373] redirect: wrap_raw
2026-07-10 15:07:36,364 INFO    MainThread:18743 [wandb_run.py:_redirect():2442] Wrapping output streams.
2026-07-10 15:07:36,364 INFO    MainThread:18743 [wandb_run.py:_redirect():2465] Redirects installed.
2026-07-10 15:07:36,366 INFO    MainThread:18743 [wandb_init.py:init():1012] run started, returning control to user process
2026-07-12 15:32:24,499 INFO    wandb-AsyncioManager-main:18743 [service_client.py:_forward_responses():122] [no run ID] Reached EOF.
2026-07-12 15:32:24,500 INFO    wandb-AsyncioManager-main:18743 [mailbox.py:close():154] [no run ID] Closing mailbox, abandoning 2 handles.
2026-07-12 15:32:24,500 ERROR   wandb-AsyncioManager-main:18743 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
    await fn()
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
    await asyncio_compat.race(
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
    async with open_task_group(race=True) as tg:
  File "/root/miniconda3/envs/genrecon/lib/python3.10/contextlib.py", line 206, in __aexit__
    await anext(self.gen)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
    await task_group._wait_all(race=race, timeout=exit_timeout)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
    raise exc
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
    result = await handle.wait_async(timeout=None)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
    response = await self._handle.wait_async(timeout=timeout)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
    raise HandleAbandonedError()
wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError
2026-07-12 15:32:25,269 ERROR   wandb-AsyncioManager-main:18743 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
    await fn()
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_stopping.py", line 87, in loop
    should_stop = await self._check_should_stop()
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/run_stopping.py", line 115, in _check_should_stop
    handle = await self._interface.deliver_async(request)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/interface/interface_sock.py", line 58, in deliver_async
    handle = await self._client.deliver(request)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 57, in deliver
    handle = self._mailbox.require_response(request)
  File "/root/miniconda3/envs/genrecon/lib/python3.10/site-packages/wandb/sdk/mailbox/mailbox.py", line 89, in require_response
    raise MailboxClosedError()
wandb.sdk.mailbox.mailbox.MailboxClosedError