transformers/pr_45649/en/quantization/selecting.md

Choose a method

If your checkpoint is already quantized, the checkpoint decides the method. Load it as shown in Load a pre-quantized checkpoint. Otherwise, pick a method based on your hardware and what you need to do with the model after loading it.

Start from your goal

Each row recommends a first method to try. Check the full comparison table before you commit to one.

Goal	Start with	Why
Fit a model in less memory with the least setup	bitsandbytes	Quantizes to 4 or 8 bits while loading without calibration, runs on CUDA, XPU, HPU, and CPU, and supports saving and fine-tuning.
Fine-tune with QLoRA	bitsandbytes	Supports 4-bit NF4 with PEFT. HQQ, EETQ, GPTQ, SINQ, and Quanto also support fine-tuning.
Speed up inference with torch.compile	torchao	generate compiles torchao models automatically with a static cache, and torchao has int4, int8, and FP8 schemes for CUDA, XPU, and CPU.
Get the best 4-bit accuracy	GPTQ	Calibrates on a dataset while loading. To use AWQ or AutoRound instead, quantize with their libraries and load the checkpoint.
Run FP8 on Ada, Hopper, or Intel XPU	Fine-grained FP8	Needs CUDA compute capability 8.9 or higher, or an XPU. FBGEMM FP8 needs compute capability 9.0 or higher, or an XPU.
Run FP4 on Blackwell	NVFP4	Quantizes to NVIDIA's 4-bit float format while loading. MXFP4 runs on compute capability 7.5 or higher, XPU, and CPU.
Run on Apple silicon	Metal	Quantizes to 2, 4, or 8 bits on MPS while loading. GGUF files also load on MPS.
Load checkpoints made with llm-compressor	compressed-tensors	Loads FP8, int4, and int8 checkpoints in the compressed-tensors format.
Compress below 4 bits	HQQ or SINQ	Quantize to 2 or 3 bits while loading without calibration. AQLM, VPTQ, and SpQR checkpoints go lower but must be quantized with their own libraries.

Show the full comparison table

Method	quant_method	Workflow	Bits	Hardware	Save	Fine-tune	Auto-compile
AQLM	aqlm	Pre-quantized only	1–2	CUDA, CPU	✅	✅ with aqlm>=1.0.2	❌
AutoRound	auto-round	Pre-quantized only	2–8	CUDA, XPU, CPU	✅	❌	❌
AWQ	awq	Pre-quantized only	4	CUDA, ROCm, XPU, CPU	✅ except ExLlama backends	✅ with gptqmodel>=5.0.0	❌
BitNet	bitnet	Pre-quantized only	1.58	CUDA	✅	Only autobitlinear in online mode	❌
bitsandbytes	bitsandbytes	While loading	4, 8	CUDA, XPU, HPU, CPU	✅	✅	❌
compressed-tensors	compressed-tensors	Pre-quantized only	Set by checkpoint	CUDA, XPU, MPS, CPU	✅	✅ except with FP8 kernels	✅
EETQ	eetq	While loading	8	CUDA	✅	✅	❌
FBGEMM FP8	fbgemm_fp8	While loading	8	CUDA (compute capability 9.0+), XPU	✅	❌	❌
Fine-grained FP8	fp8, mxfp8	While loading	8	CUDA (compute capability 8.9+), XPU	✅	❌	✅
Four Over Six	fouroversix	While loading	4	CUDA (fastest on Blackwell), CPU	✅	Only with keep_master_weights=True	❌
FP-Quant	fp_quant	While loading	4	CUDA, XPU	✅	Only with store_master_weights=True	❌
GGUF	gguf	Pre-quantized only	1–8	CUDA, MPS, XPU, CPU	❌	❌	✅
GPTQ	gptq	While loading, with calibration	2, 3, 4, 8	CUDA, ROCm, XPU, CPU	✅	✅	❌
HIGGS	higgs	While loading	2, 3, 4	CUDA	✅	❌	❌
HQQ	hqq	While loading	1, 2, 3, 4, 8	CUDA, XPU	✅	✅	❌
Metal	metal	While loading	2, 4, 8	MPS	✅	❌	❌
MXFP4	mxfp4	While loading	4	CUDA (compute capability 7.5+), XPU, CPU	✅	❌	❌
NVFP4	nvfp4	While loading	4	One Blackwell GPU	❌	❌	✅
Quanto	quanto	While loading	2, 4, 8	CUDA, MPS, XPU, CPU	❌	✅	❌
Quark	quark	Pre-quantized only	Set by checkpoint	CUDA, ROCm, CPU	❌	❌	❌
SINQ	sinq	While loading	2, 3, 4, 6, 8	One CUDA GPU, CPU	✅	✅	❌
SpQR	spqr	Pre-quantized only	3	CUDA	✅	❌	❌
torchao	torchao	While loading	4, 8	CUDA, XPU, CPU	✅	Only 8-bit schemes	✅
VPTQ	vptq	Pre-quantized only	1–8	CUDA	✅	❌	❌

Check memory with the quantization picker

The quantization picker estimates whether a model fits on your hardware at each precision, including the KV cache. Pick a model, your hardware, and what you need the method to do, and the Compare view lists the methods that match.