Hugging Face's logo

Buckets:
hf-doc-build
/
doc-dev

Files
xet
hf-doc-build/doc-dev / transformers /pr_36708 /en /exporters_quantization.md
HuggingFaceDocBuilder's picture
HuggingFaceDocBuilder
preview code
|
download
raw
12.4 kB
# Quantizing exports
Every [export](./exporters) config can take a `quantizer`, which quantizes the model during the export. The quantizers
live in `transformers.exporters.quantizers`, and each one runs at a fixed point of the export: on the `torch.export`
FX graph, before any conversion, or on the model a backend converts it into.
| Quantizer | Quantizes | Dynamo | ONNX | ExecuTorch | OpenVINO |
| --- | --- | --- | --- | --- | --- |
| [PT2EQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.PT2EQuantizer) | the FX graph, with PyTorch 2 Export (PT2E) quantization | 🟢 | 🟢 | 🟢 | 🟢 |
| [NNCFTorchFXQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.NNCFTorchFXQuantizer) | the FX graph, with [NNCF](https://github.com/openvinotoolkit/nncf) | 🟢 | 🟢 | 🟢 (weight-only) | 🟢 |
| [OnnxRuntimeQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.OnnxRuntimeQuantizer) | the ONNX model, with `onnxruntime.quantization` | 🔴 | 🟢 | 🔴 | 🔴 |
| [NNCFOnnxQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.NNCFOnnxQuantizer) | the ONNX model, with [NNCF](https://github.com/openvinotoolkit/nncf) | 🔴 | 🟢 | 🔴 | 🔴 |
| [NNCFOpenVINOQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.NNCFOpenVINOQuantizer) | the OpenVINO model, with [NNCF](https://github.com/openvinotoolkit/nncf) | 🔴 | 🔴 | 🔴 | 🟢 |
The export checks the quantizer against its backend and installed packages before tracing, so a mismatch fails early.
Like the exporters, each quantizer lists the package versions it is tested on, and warns when yours differ.
The example below quantizes a sentiment classifier to int8 with ONNX Runtime, then runs it.
```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer
from transformers.exporters import OnnxConfig, OnnxExporter, OnnxRuntimeQuantizer
model_id = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
texts = [
"A wonderful, heartfelt film.",
"The plot was dull and the acting worse.",
"I would watch it again tomorrow.",
"Two hours I will never get back.",
]
calibration_dataset = [tokenizer(text, return_tensors="pt") for text in texts]
inputs = tokenizer("An absolute delight from start to finish.", return_tensors="pt")
config = OnnxConfig(dynamic=True, quantizer=OnnxRuntimeQuantizer(calibration_dataset=calibration_dataset))
onnx_program = OnnxExporter().export(model, inputs, config)
logits = onnx_program(**inputs)[0]
print(model.config.id2label[int(logits.argmax())]) # POSITIVE
```
## PT2E
[PT2EQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.PT2EQuantizer) wraps a PT2E quantizer, from torchao or from the target runtime, and runs it
on the FX graph (`prepare_pt2e` → calibrate → `convert_pt2e`), so it works with every backend. Each PT2E quantizer
injects its own quantize/dequantize ops, so pick the one your runtime supports:
| Where you'll run | PT2E quantizer | Import from |
| --- | --- | --- |
| PyTorch inductor, ONNX Runtime (QDQ), OpenVINO | `X86InductorQuantizer` | `torchao.quantization.pt2e.quantizer.x86_inductor_quantizer` |
| ExecuTorch XNNPACK backend | `XNNPACKQuantizer` | `executorch.backends.xnnpack.quantizer.xnnpack_quantizer` |
For ExecuTorch, wrap ExecuTorch's own `XNNPACKQuantizer`:
```python
from executorch.backends.xnnpack.quantizer.xnnpack_quantizer import (
XNNPACKQuantizer,
get_symmetric_quantization_config,
)
from transformers.exporters import ExecutorchConfig, PT2EQuantizer
xnnpack = XNNPACKQuantizer().set_global(get_symmetric_quantization_config())
config = ExecutorchConfig(backend="xnnpack", quantizer=PT2EQuantizer(xnnpack, calibration_dataset=calibration_dataset))
```
Quantize generative models through `~HfExporter.export_for_generation`. Here `calibration_dataset` holds generate
kwargs, like `inputs`, and each component calibrates on the inputs captured for it. A text-only model has a `prefill`
and a `decode` component; a multimodal model splits its prefill further, into its modality encoders (such as
`image_encoder`), its projector, its `language_model` and its `lm_head` (see
[How `export_for_generation` works](./exporters#how-exportforgeneration-works)).
```python
from torchao.quantization.pt2e.quantizer.x86_inductor_quantizer import (
X86InductorQuantizer,
get_default_x86_inductor_quantization_config,
)
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.exporters import DynamoConfig, DynamoExporter, PT2EQuantizer
model_id = "meta-llama/Llama-3.2-1B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
prompts = ["The capital of France is", "Photosynthesis turns sunlight into", "The first moon landing was in"]
calibration_dataset = [tokenizer(prompt, return_tensors="pt") for prompt in prompts]
inputs = tokenizer("Once upon a time", return_tensors="pt")
x86 = X86InductorQuantizer().set_global(get_default_x86_inductor_quantization_config())
config = DynamoConfig(dynamic=True, quantizer=PT2EQuantizer(x86, calibration_dataset=calibration_dataset))
components = DynamoExporter().export_for_generation(model, inputs, config) # {"prefill": ..., "decode": ...}
```
On OpenVINO, PT2E weights stay in full precision behind quantize/dequantize pairs, which OpenVINO converts to
`FakeQuantize` and compresses to int8 itself.
## ONNX Runtime
[OnnxRuntimeQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.OnnxRuntimeQuantizer) runs `quantize_static` by default (int8 QDQ activations and weights,
calibrated on `calibration_dataset`), as in the example above, with the arguments of an
[OnnxRuntimeStaticConfig](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.OnnxRuntimeStaticConfig). With an [OnnxRuntimeDynamicConfig](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.OnnxRuntimeDynamicConfig) it runs
`quantize_dynamic` (int8 weights, activations quantized at runtime, no calibration).
```python
from transformers.exporters import OnnxRuntimeDynamicConfig, OnnxRuntimeStaticConfig
config = OnnxConfig(
quantizer=OnnxRuntimeQuantizer(OnnxRuntimeStaticConfig(per_channel=True), calibration_dataset=calibration_dataset)
)
config = OnnxConfig(quantizer=OnnxRuntimeQuantizer(OnnxRuntimeDynamicConfig()))
```
## NNCF
The three NNCF quantizers in the table above run `nncf.quantize` by default (int8 activations and weights, calibrated
on `calibration_dataset`, with `model_type=nncf.ModelType.TRANSFORMER` unless you pass another), with the arguments of
an [NNCFQuantizationConfig](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.NNCFQuantizationConfig). With an [NNCFWeightCompressionConfig](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.NNCFWeightCompressionConfig) they
run `nncf.compress_weights` (weight-only int8 or int4, with AWQ and scale estimation for int4, plus GPTQ and LoRA
correction on OpenVINO). ExecuTorch lowers only [NNCFTorchFXQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.NNCFTorchFXQuantizer)'s weight-only result.
```python
import nncf
from transformers.exporters import (
DynamoConfig,
NNCFOnnxQuantizer,
NNCFOpenVINOQuantizer,
NNCFTorchFXQuantizer,
NNCFWeightCompressionConfig,
OnnxConfig,
OpenVINOConfig,
OpenVINOExporter,
)
int4 = NNCFWeightCompressionConfig(mode=nncf.CompressWeightsMode.INT4_SYM, group_size=128)
# OpenVINO: int8 activations and weights, or weight-only int4
openvino_int8 = OpenVINOConfig(quantizer=NNCFOpenVINOQuantizer(calibration_dataset=calibration_dataset))
openvino_int4 = OpenVINOConfig(quantizer=NNCFOpenVINOQuantizer(int4))
# ONNX: weight-only int4, as ONNX Runtime's `MatMulNBits`
onnx_int4 = OnnxConfig(quantizer=NNCFOnnxQuantizer(int4))
# Dynamo: int8 on the FX graph
dynamo_int8 = DynamoConfig(quantizer=NNCFTorchFXQuantizer(calibration_dataset=calibration_dataset))
ov_model = OpenVINOExporter().export(model, inputs, openvino_int4)
```
Int8 activations run int8 matmuls, which saturate on CPUs without VNNI (AVX2-only Intel and AMD Zen 2 and older), so
expect lower accuracy there. Weight-only compression keeps the matmuls in floating point.
## Calibration
A quantizer's `calibration_dataset` is an iterable of model inputs, such as a list of tokenizer outputs or a
`DataLoader` whose batches collate to model inputs. It is a plain attribute, so you can also set it after creating
the quantizer (`quantizer.calibration_dataset = samples`). Omit it and a quantizer that needs data calibrates on the
export's own sample inputs, with a warning, because one sample can skew the observed ranges. Quantizers that need no
data (dynamic ONNX Runtime quantization, int8 weight compression) never read it.
## A different recipe per component
Pass a `{component: config}` dict to `~HfExporter.export_for_generation` to quantize each component on its own
terms. For the model above, static int8 on `prefill` and the lighter dynamic int8 on `decode`:
```python
dynamic_x86 = X86InductorQuantizer().set_global(get_default_x86_inductor_quantization_config(is_dynamic=True))
config = {
"prefill": DynamoConfig(dynamic=True, quantizer=PT2EQuantizer(x86)),
"decode": DynamoConfig(dynamic=True, quantizer=PT2EQuantizer(dynamic_x86)),
}
components = DynamoExporter().export_for_generation(model, inputs, config)
```
The dict must name every component [decompose_for_generation()](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.utils.decompose_for_generation) produces (a multimodal model adds its
encoders, and may split out `language_model` and `lm_head`). A component whose config sets no quantizer stays in full
precision. In this dict, a quantizer's `calibration_dataset` holds the component's forward kwargs, not generate
kwargs.
## Add a quantizer
Quantizers live in `transformers.exporters.quantizers`, one class per toolchain and representation. Subclass
[ExportQuantizer](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.ExportQuantizer) and declare:
- `quantizes_fx_graph`: `True` for the `torch.export` FX graph, before any conversion (so every backend takes the
result), or `False` (the default) for the model a backend converts it into (`onnx_ir.Model`, `openvino.Model`).
- `supported_formats`: the export formats it quantizes. The export rejects any other format before tracing.
- `required_packages`, and optionally `min_versions` and `tested_versions`: what it imports. The export checks
they're installed before tracing, raises below a minimum version, and warns when a version differs from the tested
one.
Then implement `quantize(model, sample_inputs, export_format)`. `model` is the FX graph or the backend model, and
`sample_inputs` are the export's own inputs. Calibrate on a [CalibrationSet](/docs/transformers/pr_36708/en/main_classes/exporters#transformers.exporters.quantizers.CalibrationSet): it reads the
quantizer's `calibration_dataset` (or `sample_inputs` when there is none) and maps each sample to the model's own
inputs with a feed from `transformers.exporters.quantizers.utils` (`traced_inputs` for the FX graph, `onnx_feed` for ONNX,
`openvino_feed` for OpenVINO).
```python
from transformers.exporters import ExportFormat
from transformers.exporters.quantizers import CalibrationSet, ExportQuantizer
from transformers.exporters.quantizers.utils import onnx_feed
class MyOnnxQuantizer(ExportQuantizer):
supported_formats = (ExportFormat.ONNX,)
required_packages = ("my_toolkit",)
tested_versions = {"my_toolkit": "1.0.0"}
def quantize(self, model, sample_inputs, export_format):
import my_toolkit
names = [graph_input.name for graph_input in model.graph.inputs]
calibration = CalibrationSet(self.calibration_dataset, sample_inputs, lambda sample: onnx_feed(names, sample))
return my_toolkit.quantize(model, calibration_feeds=list(calibration))
```

Xet Storage Details

Size:
12.4 kB
·
Xet hash:
19abba6ca77ef4a9d76a11d325310b4a5f2f6fa35ea2a381ced4465d3dd4391b

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.