object_detection/docs/README_QUANTIZATION_TOOL.md

# Table of Contents
1. [Introduction](#1)
2. [Minimal Yaml Configs](#2)
3. [Tensor Override](#3)
4. [Various Operating Modes](#4)
   - [4.1 Operating_mode: inspection](#4-1)
   - [4.2 Operating mode: full_auto](#4-2)

<details open><summary><a href="#1"><b>1. Introduction </b></a></summary><a id="1"></a>
This library contains a collection of scripts that are useful for

- improving mAp of quantized models
- designing mixed precision models
- investigation of quantization issues  

It is based on ONNX quantizer only. To have good support of 4-bits and 16-bits, it is recommended to set 'quantization.target_opset' to 21 and thus ir_version 10.
The advanced quantization services 'inspection', 'full_auto' are only enabled when 'operation_mode' is 'quantization'.
</details>

<details open><summary><a href="#2"><b>2. Minimal Yaml Configs</b></a></summary><a id="2"></a>
The legacy 'quantization' section is mandatory. The description provided in this readme assumes that we use ONNX quantizer and not LiteRT. 
Therefore, 'quantization.quantizer' must be 'Onnx_quantizer'.
If 'quantization.operating_mode' is not defined or is set to 'default', we simply run legacy quantization with the yaml specified parameters.
If some parameters are left undefined, default values are used:

      WeightSymmetric: True
      ActivationSymmetric: False
      CalibMovingAverage: False
      QuantizeBias = True
      SmoothQuant: False
      SmoothQuantAlpha: 0.5
      SmoothQuantFolding: True
      TensorQuantOverrides: None
      op_types_to_quantize = None
      nodes_to_quantize = None
      nodes_to_exclude = None
      calibrate_method = CalibrationMethod.MinMax 
      weight_type = QuantType.QInt8
      activ_type = QuantType.QInt8

Calibration moving average is still controlled in configs.quantization.onnx_extra_options parameters.
'quantization.iterative_quant_parameters' are only taken into account for 'inspection', 'full_auto'.
'iterative_quant_parameters' first defines a parameter called 'inspection_split' used in 'inspection' for defining a subset of the quantization set
on which are performed SNR measurements.
To finish, 'iterative_quant_parameters' also defines 'accuracy_tolerance'. This parameter is only used for mode 'full_auto'.
In object detection, it corresponds to the mAp margin (in percent) we tolerate relatively to full 8 bits performance.
The purpose of this selection will be more detailed in operating modes description in this readme.
</details>

<details open><summary><a href="#3"><b>3. Tensor Override</b></a></summary><a id="3"></a>
It is managed by 'quantization.onnx_extra_options.weights_tensor_override' and 'quantization.onnx_extra_options.activations_tensor_override' parameters.

Examples for setting the override parameters:

    weights_tensor_override: [['efficientnetv2-b0_1/block6a_se_reduce_1/convolution/ReadVariableOp:0', {'quant_type': Int16, 'scale': 0.0025, 'zero_point': -12}],
                              ['efficientnetv2-b0_1/stem_conv_1/convolution/merged_input:0', {'quant_type': Int16, "axis": 0}]
                             ]
    activations_tensor_override: [['efficientnetv2-b0_1/block4a_se_reduce_1/mul_1:0', {'quant_type': Int16}],
                                  ['efficientnetv2-b0_1/block4a_se_expand_1/BiasAdd__79:0', {'quant_type': Int16, 'scale': 0.002, 'zero_point': 0}]
                                 ]
Common remarks:  
'zero_point' must be:
- one integer scalar or a list of one integer value for per-tensor
- a list of several integer values for per-channel

'scale' must be: 
- one float scalar or a list of one floating-point value for per-tensor
- a list of several floating-point values for per-channel

Remarks for the weights:
- When one specifies only 'quant_type' without 'axis': whatever 'quantization.granularity' value, the tensor will be quantized on a per-tensor basis. One 'scale' and one 'zero_point'
can optionally be specified. Since weights are quantized per-tensor, bias will automatically be quantized per-tensor too.
- if 'axis' is specified, then it must be an integer: 0 if the tensor corresponds to the weights of any type of Conv, 1 for a ConvTranspose, 1 too for a Gemm or a MatMul by default.
- when 'axis' is specified (meaning that we intend to quantize the given tensor per-channel), if 'scale' and 'zero_point' are specified too, they must be a list of values whose length corresponds 
to the number of output channels for the considered tensor.

Remarks for activations:
- They are not quantized per-channel. So if 'scale' and 'zero_point' are defined, one value for each is enough.

Potential parameters conflict:
- With current onnx and onnxruntime version we must not set 'onnx_quant_parameters.nodes_to_quantize' and 'onnx_extra_options.weights_tensor_override'. It seems they are mutually exclusive.
</details>

<details open><summary><a href="#4"><b>4. Various Operating Modes</b></a></summary><a id="4"></a>

<ul><details open><summary><a href="#4-1">4.1 Operating_mode: inspection</a></summary><a id="4-1"></a>
This mode is intended to output in stm32ai_main.log the metrics for estimating quantized tensor quality whether they are weights, bias or activation tensors.
It quantizes the model with the parameters specified in the yaml sections 'quantization', including 'onnx_quant_parameters' and 'onnx_extra_options', 
which define among others bit-width and potential tensor overrides.
It has to be noticed that bias is always quantized in INT32 whatever the scheme selected in the config file. So we expect biases are high in quality.
Knowing the tensor quality, one can further investigate potential quantization issues. So it is recommended at the first time to launch this mode without
any tensor override to get the complete view of tensors quantization quality.
Important remark: computing all tensors quality metrics can take time especially if there are many layers and the quantization set is large.
To reduce the latency of this task, it is possible to use the parameter 'inspection_split' in the section 'quantization.iterative_quant_parameters' 
to only consider a split of the quantization set.
</details></ul>

<ul><details open><summary><a href="#4-2">4.2 Operating mode: full_auto</a></summary><a id="4-2"></a>
The goal of this mode is to automatically search for which weight tensors could be quantized in 4 bits or must remain in 8 bits taken into account a mAp
degradation budget of 'quantization.iterative_quant_parameters.accuracy_tolerance' defined in percent with respect to full 8 bits. 
Activations are kept in 8 bits and biases in 32 bits in any case.
It is clear that the better the weight quantizer the higher the number of weights tensors could be moved to 4-bits.
However this algorithm can also be run with the basic 'min-max' quantization algorithm embedded into ONNX.
Several steps are sequentially performed: 
      
      1. Compute float model mAp for reference
      2. Perform a first quantization with parameters in the yaml. This provides a quantized reference mAp typically W8A8
      3. Perform a second quantization with 4 bits for all weights: gives an idea how much it degrades
      4. Rank the weight tensors with respect to a composite score involving some parameters like total number of weigths and number of weights per output channels.
      5. Based on the ranking, search which weights can stay in 4 bits and which one should be on 8 bits

The score computation does not require any training step. It is done statically based on layer parameters.
Among these parameters, we can mention (not restrictive):

      a. layer type (conv2D, point-wise, depthwise, dense...)
      b. kernel size (2x2, 3x3...)
      c. total number of weights
      d. number of weights associated to each output channel

Currently, we just consider c. and d. 
In a few words, we have a 3 steps ranking procedure:

      1. First, rank in decreasing order of each tensor total number of weights
      2. To break a possible tie, rank in ascending order of tensor number of weights per output channel
      3. To break all remaining ties if any, rank in descending order of execution in the network forward pass

So, the smaller the total number of weights for a given tensor, the lower the ranking of this tensor.
To break a possible tie, we use d. considering that if there is a lot of weights per output channel for a given tensor, it is likely this tensor will be
difficult to quantize on low number of bits, with just one 'scale and one 'zero-point'. 

Once the ranking is done, we can start the search which is iterative, by dichotomy. Lower ranking tensors are first candidates for a quantization on 8 bits. 
We report the model with the minimum number of layer weights on 8 bits whose mAp is not worse than 'iterative_quant_parameters.accuracy_tolerance' compared to full 8 bits.

To run 'full_auto', and search for which weights could be quantized just on 4 bits, we need to have: 

      weight_type = Int8
      activ_type = Int8


Although, all the parameters of 'quantization' section are taken into account, we recommend not to define 'onnx_quant_parameters.op_types_to_quantize', 
'onnx_quant_parameters.nodes_to_quantize', 'onnx_quant_parameters.nodes_to_exclude' to give a change to the algorithm to explore more combinations.
Same remark for 'onnx_extra_options.weights_tensor_override' and 'onnx_extra_options.activations_tensor_override'.

On the contrary, it can be beneficial to enable SmoothQuant algorithm, because it helps balancing the quantization difficulty between the weights and the activations.
In our case, since we would like to have most of the weights on 4-bits, we need to move the quantization complexity towards the activations.
This requires a smoothing parameter 'onnx_extra_options.SmoothQuantAlpha' lower than 0.5. A typical setting would then be: 

      SmoothQuant: False
      SmoothQuantAlpha: 0.1
      SmoothQuantFolding: True

</details></ul>

</details>