| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_setup.py:_flush():80] Current SDK version is 0.23.0 |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_setup.py:_flush():80] Configure stats pid to 1318853 |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_setup.py:_flush():80] Loading settings from /home/khanhvinh9/.config/wandb/settings |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_setup.py:_flush():80] Loading settings from /home/khanhvinh9/lmc-transformer/wandb/settings |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_setup.py:_flush():80] Loading settings from environment variables |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_init.py:setup_run_log_directory():713] Logging user logs to /home/khanhvinh9/lmc-transformer/wandb/run-20251226_094108-svj9s262/logs/debug.log |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_init.py:setup_run_log_directory():714] Logging internal logs to /home/khanhvinh9/lmc-transformer/wandb/run-20251226_094108-svj9s262/logs/debug-internal.log |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_init.py:init():840] calling init triggers |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_init.py:init():845] wandb.init called with sweep_config: {} |
| config: {'_wandb': {'code_path': 'code/src/imagenet/train_model.py'}} |
| 2025-12-26 09:41:08,427 INFO MainThread:1318853 [wandb_init.py:init():888] starting backend |
| 2025-12-26 09:41:08,759 INFO MainThread:1318853 [wandb_init.py:init():891] sending inform_init request |
| 2025-12-26 09:41:08,766 INFO MainThread:1318853 [wandb_init.py:init():899] backend started and connected |
| 2025-12-26 09:41:08,768 INFO MainThread:1318853 [wandb_init.py:init():969] updated telemetry |
| 2025-12-26 09:41:08,772 INFO MainThread:1318853 [wandb_init.py:init():993] communicating run to backend with 90.0 second timeout |
| 2025-12-26 09:41:09,748 INFO MainThread:1318853 [wandb_init.py:init():1040] starting run threads in backend |
| 2025-12-26 09:41:09,895 INFO MainThread:1318853 [wandb_run.py:_console_start():2504] atexit reg |
| 2025-12-26 09:41:09,895 INFO MainThread:1318853 [wandb_run.py:_redirect():2352] redirect: wrap_raw |
| 2025-12-26 09:41:09,895 INFO MainThread:1318853 [wandb_run.py:_redirect():2421] Wrapping output streams. |
| 2025-12-26 09:41:09,895 INFO MainThread:1318853 [wandb_run.py:_redirect():2444] Redirects installed. |
| 2025-12-26 09:41:09,898 INFO MainThread:1318853 [wandb_init.py:init():1080] run started, returning control to user process |
| 2025-12-26 09:41:29,560 INFO wandb-AsyncioManager-main:1318853 [service_client.py:_forward_responses():80] Reached EOF. |
| 2025-12-26 09:41:29,560 INFO wandb-AsyncioManager-main:1318853 [mailbox.py:close():137] Closing mailbox, abandoning 1 handles. |
| 2025-12-26 09:41:29,931 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:29,938 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:29,939 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:29,944 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:29,975 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,011 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,022 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,091 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,108 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,108 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,109 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,109 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,109 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,109 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,109 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,109 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,110 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,112 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,112 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,112 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,117 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,118 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,118 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,119 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
| 2025-12-26 09:41:30,119 ERROR wandb-AsyncioManager-main:1318853 [asyncio_manager.py:fn_wrap_exceptions():183] Uncaught exception in run_soon callback. |
| Traceback (most recent call last): |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 181, in fn_wrap_exceptions |
| await fn() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 38, in publish |
| await self._send_server_request(request) |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/site-packages/wandb/sdk/lib/service/service_client.py", line 64, in _send_server_request |
| await self._writer.drain() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 371, in drain |
| await self._protocol._drain_helper() |
| File "/home/khanhvinh9/miniconda3/envs/lmc/lib/python3.10/asyncio/streams.py", line 167, in _drain_helper |
| raise ConnectionResetError('Connection lost') |
| ConnectionResetError: Connection lost |
|
|
