transformers/pr_48871/en/quantization/overview.md

Overview

Quantization lowers the memory requirements of loading and using a model by storing the weights in a lower precision while trying to preserve as much accuracy as possible. Weights are typically stored in full-precision (fp32) floating point representations, but half-precision (fp16 or bf16) are increasingly popular data types given the large size of models today. Some quantization methods can reduce the precision even further to integer representations, like int8 or int4.

Transformers supports many quantization methods, each with their pros and cons, so you can pick the best one for your specific use case. Some methods require calibration for greater accuracy and extreme compression (1-2 bits), while other methods work out of the box with on-the-fly quantization.

Use the quantization picker to find a method for your model and hardware. Pick a model, your hardware, and what you need the method to do: load the model on the fly, fine-tune it with PEFT, compile it with torch.compile(), or save the quantized result. The picker shows whether the model fits in memory at each precision, including the KV cache, and the Compare view lists the methods that match.