transformers/pr_26617/en/quantization/metal.md

Metal

Metal quantization performs affine quantization on Apple Silicon (MPS) devices using Metal kernels hosted on the Hugging Face Hub (kernels-community/mlx-quantization-metal-kernels). These kernels originate from the MLX framework and are compiled via the kernels library.

Weights are packed into uint32 tensors with per-group scales and biases, and the forward pass uses a fused dequantization + matmul Metal kernel (affine_qmm_t). This keeps memory usage low while running inference entirely on the GPU with no CPU round-trips.

Supported bit-widths are 2, 4, and 8. Group size is configurable (default 64).

Requirements

• Apple Silicon Mac (M1 / M2 / M3 / M4) with MPS support
• The kernels package:

pip install kernels

The Metal kernels are downloaded from the Hub automatically on first use — no manual compilation required.

Quantize on-the-fly

Load any model and quantize it during loading by passing a MetalConfig. All eligible nn.Linear layers are replaced with quantized versions.

from transformers import AutoModelForCausalLM, AutoTokenizer, MetalConfig

quantization_config = MetalConfig(bits=4, group_size=64)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-1B",
    device_map="mps",
    quantization_config=quantization_config,
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-1B")
inputs = tokenizer("Apple Silicon is", return_tensors="pt").to("mps")
output = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Load a pre-quantized model

If a checkpoint already contains quantized weights (weight as packed uint32, scales, qbiases), they are loaded directly — no re-quantization needed.

from transformers import AutoModelForCausalLM, MetalConfig

model = AutoModelForCausalLM.from_pretrained(
    "your-org/model-metal-4bit",
    device_map="mps",
)

Dequantize

On machines without MPS, a pre-quantized checkpoint is automatically dequantized back to float so the model remains usable on CPU or CUDA. You can also force this behavior explicitly:

from transformers import AutoModelForCausalLM, MetalConfig

config = MetalConfig(dequantize=True)
model = AutoModelForCausalLM.from_pretrained(
    "your-org/model-metal-4bit",
    quantization_config=config,
    device_map="cpu",
)

Exclude layers

Certain layers (e.g., lm_head) can be excluded from quantization via modules_to_not_convert:

config = MetalConfig(bits=4, group_size=64, modules_to_not_convert=["lm_head"])

Configuration options

Parameter	Default	Description
bits	4	Bit-width for weight quantization (2, 4, or 8)
group_size	64	Number of elements per quantization group
modules_to_not_convert	None	List of module names to keep in full precision
dequantize	False	Force dequantization to float (for non-MPS devices)