all-MiniLM-L6-v2 — ONNX INT4 & INT8 QDQ Quantized

CPU-generic quantized ONNX exports of sentence-transformers/all-MiniLM-L6-v2 that run everywhere — x86_64, ARM64/NEON, any CPU — without ISA-specific kernels.

Variants

File	Format	Size	Speed vs FP32	Cosine Similarity	Notes
`model_qint4.onnx`	INT4 MatMulNBits + GatherBlockQuantized	14 MB	~2×	0.94	Block-wise (b=32), symmetric. Uses ONNX Runtime contrib ops.
`model_qint4_qdq.onnx`	INT8 QDQ (DequantizeLinear + MatMul/Gather)	23 MB	~1×	0.966	Per-tensor uint8. Pure standard ONNX ops, no contrib.

Both variants quantize all ops — MatMul and Gather (word embeddings) — so the full model fits in CPU cache. Unlike the ISA-specific INT8 QOperator builds in the source repo (AVX2, AVX-512, ARM64), these are CPU-generic and work on any architecture with ONNX Runtime ≥ 1.20.

When to use which

model_qint4.onnx — prefer when throughput matters. Half the FP32 latency, acceptable accuracy loss from block-wise quantization.
model_qint4_qdq.onnx — prefer when embedding quality matters (e.g. semantic search, clustering). Runs at FP32 speed but preserves higher cosine similarity through per-tensor uint8 quantization of embedding lookups.

Usage

import onnxruntime as ort
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")

# INT4 (speed-first)
session = ort.InferenceSession("model_qint4.onnx")

# INT8 QDQ (accuracy-first)
# session = ort.InferenceSession("model_qint4_qdq.onnx")

tokens = tokenizer(["hello world"], return_tensors="np", padding=True, truncation=True)
outputs = session.run(None, {
    "input_ids": tokens["input_ids"],
    "attention_mask": tokens["attention_mask"],
    "token_type_ids": tokens["token_type_ids"],
})
embedding = outputs[1]  # pooled output (1 × 384)

Quantization details

Generated with ONNX Runtime's weight-only quantization tools:

model_qint4.onnx: MatMulNBitsQuantizer with block_size=32, is_symmetric=True, accuracy_level=4, op_types_to_quantize=("MatMul", "Gather"). Produces MatMulNBits and GatherBlockQuantized ops (contrib, opset 21).
model_qint4_qdq.onnx: quantize_dynamic with weight_type=QuantType.QUInt8, op_types_to_quantize=["MatMul", "Gather"]. Wraps MatMul/Gather in standard DequantizeLinear/QuantizeLinear pairs.

Reproducing

Both variants are generated from the FP32 model.onnx in sentence-transformers/all-MiniLM-L6-v2.

# Install dependencies
pip install onnx onnxruntime

# Download source + generate both quantized variants
./generate.sh

# Or generate and upload to Hugging Face
# ./generate.sh --upload

Downloads last month: -