optimum-intel/pr_1503/en/neural_compressor/reference.md

# Reference

## INCQuantizer[[optimum.intel.INCQuantizer]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCQuantizer</name><anchor>optimum.intel.INCQuantizer</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/quantization.py#L78</source><parameters>[{"name": "model", "val": ": typing.Union[transformers.modeling_utils.PreTrainedModel, torch.nn.modules.module.Module]"}, {"name": "eval_fn", "val": ": typing.Optional[typing.Callable[[transformers.modeling_utils.PreTrainedModel], int]] = None"}, {"name": "calibration_fn", "val": ": typing.Optional[typing.Callable[[transformers.modeling_utils.PreTrainedModel], int]] = None"}, {"name": "task", "val": ": typing.Optional[str] = None"}, {"name": "seed", "val": ": int = 42"}]</parameters></docstring>

Handle the Neural Compressor quantization process.



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>get_calibration_dataset</name><anchor>optimum.intel.INCQuantizer.get_calibration_dataset</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/quantization.py#L247</source><parameters>[{"name": "dataset_name", "val": ": str"}, {"name": "num_samples", "val": ": int = 100"}, {"name": "dataset_config_name", "val": ": typing.Optional[str] = None"}, {"name": "dataset_split", "val": ": str = 'train'"}, {"name": "preprocess_function", "val": ": typing.Optional[typing.Callable] = None"}, {"name": "preprocess_batch", "val": ": bool = True"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "token", "val": ": typing.Union[bool, str, NoneType] = None"}]</parameters><paramsdesc>- **dataset_name** (`str`) --
  The dataset repository name on the Hugging Face Hub or path to a local directory containing data files
  in generic formats and optionally a dataset script, if it requires some code to read the data files.
- **num_samples** (`int`, defaults to 100) --
  The maximum number of samples composing the calibration dataset.
- **dataset_config_name** (`str`, *optional*) --
  The name of the dataset configuration.
- **dataset_split** (`str`, defaults to `"train"`) --
  Which split of the dataset to use to perform the calibration step.
- **preprocess_function** (`Callable`, *optional*) --
  Processing function to apply to each example after loading dataset.
- **preprocess_batch** (`bool`, defaults to `True`) --
  Whether the `preprocess_function` should be batched.
- **use_auth_token** (Optional[Union[bool, str]], defaults to `None`) --
  Deprecated. Please use `token` instead.
- **token** (Optional[Union[bool, str]], defaults to `None`) --
  The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated
  when running `huggingface-cli login` (stored in `~/.huggingface`).</paramsdesc><paramgroups>0</paramgroups><retdesc>The calibration `datasets.Dataset` to use for the post-training static quantization calibration step.</retdesc></docstring>

Create the calibration `datasets.Dataset` to use for the post-training static quantization calibration step.






</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>quantize</name><anchor>optimum.intel.INCQuantizer.quantize</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/quantization.py#L120</source><parameters>[{"name": "quantization_config", "val": ": ForwardRef('PostTrainingQuantConfig')"}, {"name": "save_directory", "val": ": typing.Union[str, pathlib.Path]"}, {"name": "calibration_dataset", "val": ": Dataset = None"}, {"name": "batch_size", "val": ": int = 8"}, {"name": "data_collator", "val": ": typing.Optional[transformers.data.data_collator.DataCollator] = None"}, {"name": "remove_unused_columns", "val": ": bool = True"}, {"name": "file_name", "val": ": str = None"}, {"name": "**kwargs", "val": ""}]</parameters><paramsdesc>- **quantization_config** (`Union[PostTrainingQuantConfig]`) --
  The configuration containing the parameters related to quantization.
- **save_directory** (`Union[str, Path]`) --
  The directory where the quantized model should be saved.
- **calibration_dataset** (`datasets.Dataset`, defaults to `None`) --
  The dataset to use for the calibration step, needed for post-training static quantization.
- **batch_size** (`int`, defaults to 8) --
  The number of calibration samples to load per batch.
- **data_collator** (`DataCollator`, defaults to `None`) --
  The function to use to form a batch from a list of elements of the calibration dataset.
- **remove_unused_columns** (`bool`, defaults to `True`) --
  Whether or not to remove the columns unused by the model forward method.</paramsdesc><paramgroups>0</paramgroups></docstring>

Quantize a model given the optimization specifications defined in `quantization_config`.




</div></div>

## INCTrainer[[optimum.intel.INCTrainer]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCTrainer</name><anchor>optimum.intel.INCTrainer</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L109</source><parameters>[{"name": "model", "val": ": typing.Union[transformers.modeling_utils.PreTrainedModel, torch.nn.modules.module.Module] = None"}, {"name": "args", "val": ": TrainingArguments = None"}, {"name": "data_collator", "val": ": typing.Optional[transformers.data.data_collator.DataCollator] = None"}, {"name": "train_dataset", "val": ": typing.Optional[torch.utils.data.dataset.Dataset] = None"}, {"name": "eval_dataset", "val": ": typing.Optional[torch.utils.data.dataset.Dataset] = None"}, {"name": "processing_class", "val": ": typing.Union[transformers.tokenization_utils_base.PreTrainedTokenizerBase, transformers.feature_extraction_utils.FeatureExtractionMixin, NoneType] = None"}, {"name": "model_init", "val": ": typing.Callable[[], transformers.modeling_utils.PreTrainedModel] = None"}, {"name": "compute_loss_func", "val": ": typing.Optional[typing.Callable] = None"}, {"name": "compute_metrics", "val": ": typing.Optional[typing.Callable[[transformers.trainer_utils.EvalPrediction], typing.Dict]] = None"}, {"name": "callbacks", "val": ": typing.Optional[typing.List[transformers.trainer_callback.TrainerCallback]] = None"}, {"name": "optimizers", "val": ": typing.Tuple[torch.optim.optimizer.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None)"}, {"name": "preprocess_logits_for_metrics", "val": ": typing.Callable[[torch.Tensor, torch.Tensor], torch.Tensor] = None"}, {"name": "quantization_config", "val": ": typing.Optional[neural_compressor.config._BaseQuantizationConfig] = None"}, {"name": "pruning_config", "val": ": typing.Optional[neural_compressor.config._BaseQuantizationConfig] = None"}, {"name": "distillation_config", "val": ": typing.Optional[neural_compressor.config._BaseQuantizationConfig] = None"}, {"name": "task", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>

INCTrainer enables Intel Neural Compression quantization aware training, pruning and distillation.



<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>compute_distillation_loss</name><anchor>optimum.intel.INCTrainer.compute_distillation_loss</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L843</source><parameters>[{"name": "student_outputs", "val": ""}, {"name": "teacher_outputs", "val": ""}]</parameters></docstring>

How the distillation loss is computed given the student and teacher outputs.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>compute_loss</name><anchor>optimum.intel.INCTrainer.compute_loss</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L767</source><parameters>[{"name": "model", "val": ""}, {"name": "inputs", "val": ""}, {"name": "return_outputs", "val": " = False"}, {"name": "num_items_in_batch", "val": " = None"}]</parameters></docstring>

How the loss is computed by Trainer. By default, all models return the loss in the first element.


</div>
<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>save_model</name><anchor>optimum.intel.INCTrainer.save_model</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L676</source><parameters>[{"name": "output_dir", "val": ": typing.Optional[str] = None"}, {"name": "_internal_call", "val": ": bool = False"}]</parameters></docstring>

Will save the model, so you can reload it using `from_pretrained()`.
Will only save from the main process.


</div></div>

## INCModel[[optimum.intel.INCModel]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModel</name><anchor>optimum.intel.INCModel</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L71</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForSequenceClassification[[optimum.intel.INCModelForSequenceClassification]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForSequenceClassification</name><anchor>optimum.intel.INCModelForSequenceClassification</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L396</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForQuestionAnswering[[optimum.intel.INCModelForQuestionAnswering]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForQuestionAnswering</name><anchor>optimum.intel.INCModelForQuestionAnswering</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L391</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForTokenClassification[[optimum.intel.INCModelForTokenClassification]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForTokenClassification</name><anchor>optimum.intel.INCModelForTokenClassification</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L401</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForMultipleChoice[[optimum.intel.INCModelForMultipleChoice]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForMultipleChoice</name><anchor>optimum.intel.INCModelForMultipleChoice</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L406</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForMaskedLM[[optimum.intel.INCModelForMaskedLM]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForMaskedLM</name><anchor>optimum.intel.INCModelForMaskedLM</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L416</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForCausalLM[[optimum.intel.INCModelForCausalLM]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForCausalLM</name><anchor>optimum.intel.INCModelForCausalLM</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L426</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>

## INCModelForSeq2SeqLM[[optimum.intel.INCModelForSeq2SeqLM]]

<div class="docstring border-l-2 border-t-2 pl-4 pt-3.5 border-gray-100 rounded-tl-xl mb-6 mt-8">


<docstring><name>class optimum.intel.INCModelForSeq2SeqLM</name><anchor>optimum.intel.INCModelForSeq2SeqLM</anchor><source>https://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L411</source><parameters>[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]</parameters></docstring>


</div>