runtime error

Exit code: 1. Reason: ��▋ | 6/9 [00:02<00:01, 2.86it/s] Loading checkpoint shards: 100%|██████████| 9/9 [00:03<00:00, 2.88it/s] Loading checkpoint shards: 100%|██████████| 9/9 [00:03<00:00, 2.88it/s] Loading pipeline components...: 50%|█████ | 3/6 [00:05<00:05, 1.90s/it] Loading pipeline components...: 100%|██████████| 6/6 [00:06<00:00, 1.08s/it] SPACES_ZERO_GPU_DEBUG self.arg_queue._writer.fileno()=11 SPACES_ZERO_GPU_DEBUG self.res_queue._writer.fileno()=13 Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 154, in worker_init torch.move(callback=callback) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 450, in move e.submit(copy_context().run, _move, callback=callback).result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 433, in _move original_cuda = original.pin_memory().cuda(non_blocking=True) RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1123, please report a bug to PyTorch. Traceback (most recent call last): File "/home/user/app/app.py", line 151, in <module> optimize_pipeline_(pipe, image=Image.new("RGB", (1024, 1024)), prompt="prompt") File "/home/user/app/optimization.py", line 70, in optimize_pipeline_ spaces.aoti_apply(compile_transformer(), pipeline.transformer) File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 238, in gradio_handler raise error("ZeroGPU worker error", res.error_cls) gradio.exceptions.Error: 'RuntimeError'

Container logs: