optimum-intel/pr_1503/en/neural_compressor/reference.md

Reference

INCQuantizer[[optimum.intel.INCQuantizer]]

class optimum.intel.INCQuantizeroptimum.intel.INCQuantizerhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/quantization.py#L78[{"name": "model", "val": ": typing.Union[transformers.modeling_utils.PreTrainedModel, torch.nn.modules.module.Module]"}, {"name": "eval_fn", "val": ": typing.Optional[typing.Callable[[transformers.modeling_utils.PreTrainedModel], int]] = None"}, {"name": "calibration_fn", "val": ": typing.Optional[typing.Callable[[transformers.modeling_utils.PreTrainedModel], int]] = None"}, {"name": "task", "val": ": typing.Optional[str] = None"}, {"name": "seed", "val": ": int = 42"}]

Handle the Neural Compressor quantization process.

get_calibration_datasetoptimum.intel.INCQuantizer.get_calibration_datasethttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/quantization.py#L247[{"name": "dataset_name", "val": ": str"}, {"name": "num_samples", "val": ": int = 100"}, {"name": "dataset_config_name", "val": ": typing.Optional[str] = None"}, {"name": "dataset_split", "val": ": str = 'train'"}, {"name": "preprocess_function", "val": ": typing.Optional[typing.Callable] = None"}, {"name": "preprocess_batch", "val": ": bool = True"}, {"name": "use_auth_token", "val": ": typing.Union[bool, str, NoneType] = None"}, {"name": "token", "val": ": typing.Union[bool, str, NoneType] = None"}]- dataset_name (str) -- The dataset repository name on the Hugging Face Hub or path to a local directory containing data files in generic formats and optionally a dataset script, if it requires some code to read the data files.

• num_samples (int, defaults to 100) -- The maximum number of samples composing the calibration dataset.
• dataset_config_name (str, optional) -- The name of the dataset configuration.
• dataset_split (str, defaults to "train") -- Which split of the dataset to use to perform the calibration step.
• preprocess_function (Callable, optional) -- Processing function to apply to each example after loading dataset.
• preprocess_batch (bool, defaults to True) -- Whether the preprocess_function should be batched.
• use_auth_token (Optional[Union[bool, str]], defaults to None) -- Deprecated. Please use token instead.
• token (Optional[Union[bool, str]], defaults to None) -- The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running huggingface-cli login (stored in ~/.huggingface).0The calibration datasets.Dataset to use for the post-training static quantization calibration step.

Create the calibration datasets.Dataset to use for the post-training static quantization calibration step.

quantizeoptimum.intel.INCQuantizer.quantizehttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/quantization.py#L120[{"name": "quantization_config", "val": ": ForwardRef('PostTrainingQuantConfig')"}, {"name": "save_directory", "val": ": typing.Union[str, pathlib.Path]"}, {"name": "calibration_dataset", "val": ": Dataset = None"}, {"name": "batch_size", "val": ": int = 8"}, {"name": "data_collator", "val": ": typing.Optional[transformers.data.data_collator.DataCollator] = None"}, {"name": "remove_unused_columns", "val": ": bool = True"}, {"name": "file_name", "val": ": str = None"}, {"name": "**kwargs", "val": ""}]- quantization_config (Union[PostTrainingQuantConfig]) -- The configuration containing the parameters related to quantization.

• save_directory (Union[str, Path]) -- The directory where the quantized model should be saved.
• calibration_dataset (datasets.Dataset, defaults to None) -- The dataset to use for the calibration step, needed for post-training static quantization.
• batch_size (int, defaults to 8) -- The number of calibration samples to load per batch.
• data_collator (DataCollator, defaults to None) -- The function to use to form a batch from a list of elements of the calibration dataset.
• remove_unused_columns (bool, defaults to True) -- Whether or not to remove the columns unused by the model forward method.0

Quantize a model given the optimization specifications defined in quantization_config.

INCTrainer[[optimum.intel.INCTrainer]]

class optimum.intel.INCTraineroptimum.intel.INCTrainerhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L109[{"name": "model", "val": ": typing.Union[transformers.modeling_utils.PreTrainedModel, torch.nn.modules.module.Module] = None"}, {"name": "args", "val": ": TrainingArguments = None"}, {"name": "data_collator", "val": ": typing.Optional[transformers.data.data_collator.DataCollator] = None"}, {"name": "train_dataset", "val": ": typing.Optional[torch.utils.data.dataset.Dataset] = None"}, {"name": "eval_dataset", "val": ": typing.Optional[torch.utils.data.dataset.Dataset] = None"}, {"name": "processing_class", "val": ": typing.Union[transformers.tokenization_utils_base.PreTrainedTokenizerBase, transformers.feature_extraction_utils.FeatureExtractionMixin, NoneType] = None"}, {"name": "model_init", "val": ": typing.Callable[[], transformers.modeling_utils.PreTrainedModel] = None"}, {"name": "compute_loss_func", "val": ": typing.Optional[typing.Callable] = None"}, {"name": "compute_metrics", "val": ": typing.Optional[typing.Callable[[transformers.trainer_utils.EvalPrediction], typing.Dict]] = None"}, {"name": "callbacks", "val": ": typing.Optional[typing.List[transformers.trainer_callback.TrainerCallback]] = None"}, {"name": "optimizers", "val": ": typing.Tuple[torch.optim.optimizer.Optimizer, torch.optim.lr_scheduler.LambdaLR] = (None, None)"}, {"name": "preprocess_logits_for_metrics", "val": ": typing.Callable[[torch.Tensor, torch.Tensor], torch.Tensor] = None"}, {"name": "quantization_config", "val": ": typing.Optional[neural_compressor.config._BaseQuantizationConfig] = None"}, {"name": "pruning_config", "val": ": typing.Optional[neural_compressor.config._BaseQuantizationConfig] = None"}, {"name": "distillation_config", "val": ": typing.Optional[neural_compressor.config._BaseQuantizationConfig] = None"}, {"name": "task", "val": ": typing.Optional[str] = None"}, {"name": "**kwargs", "val": ""}]

INCTrainer enables Intel Neural Compression quantization aware training, pruning and distillation.

compute_distillation_lossoptimum.intel.INCTrainer.compute_distillation_losshttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L843[{"name": "student_outputs", "val": ""}, {"name": "teacher_outputs", "val": ""}]

How the distillation loss is computed given the student and teacher outputs.

compute_lossoptimum.intel.INCTrainer.compute_losshttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L767[{"name": "model", "val": ""}, {"name": "inputs", "val": ""}, {"name": "return_outputs", "val": " = False"}, {"name": "num_items_in_batch", "val": " = None"}]

How the loss is computed by Trainer. By default, all models return the loss in the first element.

save_modeloptimum.intel.INCTrainer.save_modelhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/trainer.py#L676[{"name": "output_dir", "val": ": typing.Optional[str] = None"}, {"name": "_internal_call", "val": ": bool = False"}]

Will save the model, so you can reload it using from_pretrained(). Will only save from the main process.

INCModel[[optimum.intel.INCModel]]

class optimum.intel.INCModeloptimum.intel.INCModelhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L71[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForSequenceClassification[[optimum.intel.INCModelForSequenceClassification]]

class optimum.intel.INCModelForSequenceClassificationoptimum.intel.INCModelForSequenceClassificationhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L396[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForQuestionAnswering[[optimum.intel.INCModelForQuestionAnswering]]

class optimum.intel.INCModelForQuestionAnsweringoptimum.intel.INCModelForQuestionAnsweringhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L391[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForTokenClassification[[optimum.intel.INCModelForTokenClassification]]

class optimum.intel.INCModelForTokenClassificationoptimum.intel.INCModelForTokenClassificationhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L401[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForMultipleChoice[[optimum.intel.INCModelForMultipleChoice]]

class optimum.intel.INCModelForMultipleChoiceoptimum.intel.INCModelForMultipleChoicehttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L406[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForMaskedLM[[optimum.intel.INCModelForMaskedLM]]

class optimum.intel.INCModelForMaskedLMoptimum.intel.INCModelForMaskedLMhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L416[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForCausalLM[[optimum.intel.INCModelForCausalLM]]

class optimum.intel.INCModelForCausalLMoptimum.intel.INCModelForCausalLMhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L426[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]

INCModelForSeq2SeqLM[[optimum.intel.INCModelForSeq2SeqLM]]

class optimum.intel.INCModelForSeq2SeqLMoptimum.intel.INCModelForSeq2SeqLMhttps://github.com/huggingface/optimum-intel/blob/vr_1503/optimum/intel/neural_compressor/modeling_base.py#L411[{"name": "model", "val": ""}, {"name": "config", "val": ": PretrainedConfig = None"}, {"name": "model_save_dir", "val": ": typing.Union[str, pathlib.Path, tempfile.TemporaryDirectory, NoneType] = None"}, {"name": "q_config", "val": ": typing.Dict = None"}, {"name": "inc_config", "val": ": typing.Dict = None"}, {"name": "**kwargs", "val": ""}]