| ---
|
| language: en
|
| license: apache-2.0
|
| base_model: Vandita/Bert-finetuned-Sarc
|
| tags: [sarcasm-detection, onnx, quantization, int8, static-quantization, emoji]
|
| pipeline_tag: text-classification
|
| ---
|
|
|
| # QuantizedSarcBERT-static-v2
|
|
|
| INT8 **statically** quantized ONNX build of [`Vandita/Bert-finetuned-Sarc`](https://huggingface.co/Vandita/Bert-finetuned-Sarc),
|
| for CPU inference.
|
|
|
| ## Padding invariance — read this before serving
|
|
|
| Activation scales are **frozen at calibration time**, so they cannot change with input
|
| shape. Measured on this build:
|
|
|
| | variant | pad-sensitivity | mixed-batch | safe to batch? |
|
| |---|---|---|---|
|
| | fp32 (reference) | 1.667e-09 | 3.904e-08 | yes |
|
| | INT8 **dynamic** PTQ | 8.816e-01 | 9.173e-01 | **no — serve unpadded only** |
|
| | **INT8 static (this model)** | **0.000e+00** | **0.000e+00** | **yes** |
|
|
|
| *pad-sensitivity* = max change in P(sarcastic) for the same text alone vs padded, at batch
|
| size 1. *mixed-batch* = same text alone vs inside a mixed-length batch.
|
|
|
| Dynamic PTQ recomputes activation scales from each tensor's observed range at inference
|
| time. Padded positions enter that range, so they change the scale, so they change how the
|
| **real** tokens are quantized — the attention mask stops padded positions contributing to
|
| attention, but not to the quantization range. Static quantization removes the dependence
|
| entirely.
|
|
|
| ## Results
|
|
|
|
|
| **SarcOjiTest1**
|
|
|
| | variant | accuracy | precision | recall | F1 | MCC | ROC-AUC |
|
| |---|---|---|---|---|---|---|
|
| | fp32 (original) | 0.6605 | 0.7027 | 0.6143 | 0.6555 | 0.3266 | 0.7135 |
|
| | INT8 static (this model) | 0.6507 | 0.6842 | 0.6235 | 0.6524 | 0.3041 | 0.7040 |
|
|
|
| **SarcOjiTest2**
|
|
|
| | variant | accuracy | precision | recall | F1 | MCC | ROC-AUC |
|
| |---|---|---|---|---|---|---|
|
| | fp32 (original) | 0.7124 | 0.4488 | 0.6635 | 0.5355 | 0.3518 | 0.7419 |
|
| | INT8 static (this model) | 0.6990 | 0.4328 | 0.6603 | 0.5229 | 0.3317 | 0.7417 |
|
|
|
| SarcOjiTest2 is ~75/25 negative-skewed; MCC and ROC-AUC are the meaningful columns there.
|
|
|
| ## Performance (x86 CPU, single thread, batch-of-1)
|
|
|
| | variant | p50 (ms) | p95 (ms) | size (MB) |
|
| |---|---|---|---|
|
| | fp32 | 67.95 | 90.32 | 437.6 |
|
| | INT8 static | 29.04 | 52.28 | 183.5 |
|
|
|
| Not an on-device ARM measurement.
|
|
|
| ## Quantization recipe
|
|
|
| - Static PTQ, QDQ format, per-channel weights
|
| - Calibration: `percentile` on 128
|
| in-domain SarcOji samples, padded as deployed
|
| - Ops quantized: MatMul and Gemm only (where BERT's compute is; leaving
|
| Add/LayerNorm/Softmax in float preserves accuracy at negligible speed cost)
|
| - Exported with eager attention, dynamic batch and sequence axes, opset 17
|
|
|
| ## Usage
|
|
|
| ```python
|
| from optimum.onnxruntime import ORTModelForSequenceClassification
|
| from transformers import AutoTokenizer
|
|
|
| tok = AutoTokenizer.from_pretrained("Vandita/QuantizedSarcBERT-static-v2")
|
| model = ORTModelForSequenceClassification.from_pretrained(
|
| "Vandita/QuantizedSarcBERT-static-v2", file_name="model_int8_static.onnx")
|
|
|
| inputs = tok(["Oh great, another meeting."], return_tensors="pt", padding=True)
|
| logits = model(**inputs).logits # batching is safe on this build
|
| ```
|
|
|
