This is the official MatGPTQ checkpoint of meta-llama/Llama-3.1-8B-Instruct, produced as described in the "MatGPTQ: Accurate and Efficient Post-Training Matryoshka Quantization" paper.
This model can be run via vLLM. Checkout our integration at IST-DASLab/MatGPTQ