2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_setup.py:_flush():80] Current SDK version is 0.23.0
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_setup.py:_flush():80] Configure stats pid to 1318853
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_setup.py:_flush():80] Loading settings from /home/khanhvinh9/.config/wandb/settings
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_setup.py:_flush():80] Loading settings from /home/khanhvinh9/lmc-transformer/wandb/settings
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_setup.py:_flush():80] Loading settings from environment variables
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_init.py:setup_run_log_directory():713] Logging user logs to /home/khanhvinh9/lmc-transformer/wandb/run-20251226_094108-svj9s262/logs/debug.log
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_init.py:setup_run_log_directory():714] Logging internal logs to /home/khanhvinh9/lmc-transformer/wandb/run-20251226_094108-svj9s262/logs/debug-internal.log
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_init.py:init():840] calling init triggers
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_init.py:init():845] wandb.init called with sweep_config: {}
config: {'_wandb': {'code_path': 'code/src/imagenet/train_model.py'}}
2025-12-26 09:41:08,427 INFO    MainThread:1318853 [wandb_init.py:init():888] starting backend
2025-12-26 09:41:08,759 INFO    MainThread:1318853 [wandb_init.py:init():891] sending inform_init request
2025-12-26 09:41:08,766 INFO    MainThread:1318853 [wandb_init.py:init():899] backend started and connected
2025-12-26 09:41:08,768 INFO    MainThread:1318853 [wandb_init.py:init():969] updated telemetry
2025-12-26 09:41:08,772 INFO    MainThread:1318853 [wandb_init.py:init():993] communicating run to backend with 90.0 second timeout
2025-12-26 09:41:09,748 INFO    MainThread:1318853 [wandb_init.py:init():1040] starting run threads in backend
2025-12-26 09:41:09,895 INFO    MainThread:1318853 [wandb_run.py:_console_start():2504] atexit reg
2025-12-26 09:41:09,895 INFO    MainThread:1318853 [wandb_run.py:_redirect():2352] redirect: wrap_raw
2025-12-26 09:41:09,895 INFO    MainThread:1318853 [wandb_run.py:_redirect():2421] Wrapping output streams.
2025-12-26 09:41:09,895 INFO    MainThread:1318853 [wandb_run.py:_redirect():2444] Redirects installed.
2025-12-26 09:41:09,898 INFO    MainThread:1318853 [wandb_init.py:init():1080] run started, returning control to user process
2025-12-26 09:41:29,560 INFO    wandb-AsyncioManager-main:1318853 [service_client.py:_forward_responses():80] Reached EOF.
2025-12-26 09:41:29,560 INFO    wandb-AsyncioManager-main:1318853 [mailbox.py:close():137] Closing mailbox, abandoning 1 handles.
2025-12-26 09:41:29,931 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:29,938 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:29,939 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:29,944 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:29,975 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,011 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,022 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,091 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,108 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,108 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,109 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,109 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,109 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,109 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,109 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,109 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,110 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,112 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,112 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,112 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,117 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,118 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,118 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,119 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
2025-12-26 09:41:30,119 ERROR   wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback.
Traceback (most recent call last):
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions
    await fn()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish
    await self._send_server_request(request)
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request
    await self._writer.drain()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain
    await self._protocol._drain_helper()
  File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost