runtime error

Exit code: 1. Reason: 0:00<?, ?B/s] tokenizer_config.json: 100%|██████████| 61.8k/61.8k [00:00<00:00, 253MB/s] Loading pipeline components...: 0%| | 0/6 [00:00<?, ?it/s]`torch_dtype` is deprecated! Use `dtype` instead! Loading checkpoint shards: 0%| | 0/3 [00:00<?, ?it/s] Loading checkpoint shards: 100%|██████████| 3/3 [00:00<00:00, 13.01it/s] Loading pipeline components...: 100%|██████████| 6/6 [00:00<00:00, 7.87it/s] SPACES_ZERO_GPU_DEBUG self.arg_queue._writer.fileno()=10 SPACES_ZERO_GPU_DEBUG self.res_queue._writer.fileno()=14 Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 152, in worker_init torch.move(callback=callback) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 447, in move e.submit(copy_context().run, _move, callback=callback).result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 458, in result return self.__get_result() File "/usr/local/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result raise self._exception File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/usr/local/lib/python3.10/site-packages/spaces/zero/torch/patching.py", line 430, in _move original_cuda = original.pin_memory().cuda(non_blocking=True) RuntimeError: NVML_SUCCESS == r INTERNAL ASSERT FAILED at "/pytorch/c10/cuda/CUDACachingAllocator.cpp":1131, please report a bug to PyTorch. Traceback (most recent call last): File "/home/user/app/app.py", line 55, in <module> optimize_pipeline_(pipe, File "/home/user/app/optimization.py", line 112, in optimize_pipeline_ cl1, cl2, cp1, cp2 = compile_transformer() File "/usr/local/lib/python3.10/site-packages/spaces/zero/wrappers.py", line 227, in gradio_handler raise error("ZeroGPU worker error", res.error_cls) gradio.exceptions.Error: 'RuntimeError'

Container logs: