| --- |
| pipeline_tag: question-answering |
| datasets: |
| - squad |
| metrics: |
| - exact_match |
| - f1 |
| tags: |
| - distilbert |
| --- |
| |
| **task**: `question-answering` |
|
|
| Fixed parameters: |
| * **model_name_or_path**: `distilbert-base-uncased-distilled-squad` |
| * **dataset**: |
| * **path**: `squad` |
| * **name**: `None` |
| * **calibration_split**: `train` |
| * **eval_split**: `validation` |
| * **data_keys**: `{'question': 'question', 'context': 'context'}` |
| * **ref_keys**: `['answers']` |
| * **max_seq_length**: `128` |
| * **node_exclusion**: `[]` |
| * **per_channel**: `False` |
| * **calibration**: |
| * **method**: `minmax` |
| * **num_calibration_samples**: `20` |
| * **calibration_histogram_percentile**: `None` |
| * **calibration_moving_average**: `None` |
| * **calibration_moving_average_constant**: `None` |
| * **framework**: `onnxruntime` |
| * **framework_args**: |
| * **opset**: `15` |
| * **optimization_level**: `1` |
| * **aware_training**: `False` |
| |
| Benchmarked parameters: |
| * **quantization_approach**: `dynamic`, `static` |
| * **operators_to_quantize**: `['Add', 'MatMul']`, `['Add']` |
|
|
| ## Evaluation |
| Below, time metrics for |
| * Batch size: 8 |
| * Input length: 128 |
| | quantization_approach | operators_to_quantize | | latency_mean (original, ms) | latency_mean (optimized, ms) | | throughput (original, /s) | throughput (optimized, /s) | | exact_match (original) | exact_match (optimized) | | f1 (original) | f1 (optimized) | |
| | :-------------------: | :-------------------: | :-: | :-------------------------: | :--------------------------: | :-: | :-----------------------: | :------------------------: | :-: | :--------------------: | :---------------------: | :-: | :-----------: | :------------: | |
| | `dynamic` | `['Add', 'MatMul']` | \| | 247.65 | 75.80 | \| | 4.50 | 13.50 | \| | 90.000 | 85.000 | \| | 90.000 | 86.667 | |
| | `dynamic` | `['Add']` | \| | 234.61 | 191.78 | \| | 4.50 | 5.50 | \| | 90.000 | 90.000 | \| | 90.000 | 90.000 | |
| | `static` | `['Add', 'MatMul']` | \| | 238.04 | 131.81 | \| | 4.50 | 8.00 | \| | 90.000 | 50.000 | \| | 90.000 | 59.362 | |
| | `static` | `['Add']` | \| | 241.84 | 257.85 | \| | 4.50 | 4.00 | \| | 90.000 | 70.000 | \| | 90.000 | 76.333 | |
| |
