see if this helps: https://huggingface.co/mistralai/Mistral-7B-v0.1/discussions/23#6518110742097d8c5907f822

@harpercarroll What is the minimum requirement of System RAM, GPU RAM to inference the bitsandbytes nf4 ,use_4bit = True train model. I train model of bnb_4bit_quant_type = "nf4" after training model size is same like 15GB. How we can reduce model Size ?

This will help: https://medium.com/@qendelai/fine-tuning-mistral-7b-instruct-model-in-colab-a-beginners-guide-0f7bebccf11c

hey @Tapendra you might find Impulse AI (https://www.impulselabs.ai/) useful. we make it super easy to fine-tune and deploy open source models. hopefully you find it helpful! i know not relevant to your problem above but might be easier to use us to fine tune and deploy

docs: https://docs.impulselabs.ai/introduction
python sdk: https://pypi.org/project/impulse-api-sdk-python/