[Bug] I have a error

#75

by Arigadam - opened Jan 11, 2024

Jan 11, 2024

I have a error:

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 76.00 MiB. GPU 0 has a total capacty of 8.00 GiB of which 0 bytes is free. Of the allocated memory 7.22 GiB is allocated by PyTorch, and 43.56 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

How to fix?

joey00072

Jan 11, 2024

pass load_in_4bit=True to lorad_pretrained

gugarosa

Microsoft org Jan 12, 2024

Hello @Arigadam ! Please use any quantization approach or try lowering the micro batch-size.

gugarosa changed discussion status to closed Jan 12, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment