README.md

metadata

license: apache-2.0
language:
  - multilingual
tags:
  - cross-encoder
  - reranker
  - feed-ranking
  - onnx
  - int8
pipeline_tag: text-classification
base_model: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1
library_name: optimum

Third-Pass Feed Ranker — ONNX + INT8

ONNX and dynamically-quantized INT8 builds of the third-pass feed ranker for fast CPU inference. Same model, two files:

file	precision	size	notes
model.onnx	fp32	~471 MB	scores are identical to the PyTorch model
model_quantized.onnx	int8 (dynamic)	~118 MB	4× smaller; scores within ~0.05 of fp32 (ranking order preserved)

CPU throughput (job_title × post scoring)

Measured on a desktop CPU (AVX2/AVX-VNNI, no AVX-512), 8 threads:

build	items/sec
onnx fp32	~910
onnx int8	1080 (1.2×)

On server CPUs with AVX-512 VNNI (e.g. Intel Xeon Scalable), the INT8 speedup is typically 2–4× — this desktop under-shows it. Absolute rates depend on post length.

Usage

from optimum.onnxruntime import ORTModelForSequenceClassification
from transformers import AutoTokenizer
import torch

name = "you/third-pass-feed-ranker-onnx"
tok = AutoTokenizer.from_pretrained(name)
model = ORTModelForSequenceClassification.from_pretrained(name, file_name="model_quantized.onnx")  # or model.onnx

title, posts = "Registered Nurse", ["Updated sepsis screening pathway is now live.", "Q3 revenue beat expectations!"]
enc = tok([title]*len(posts), posts, truncation=True, max_length=160, padding=True, return_tensors="pt")
scores = model(**enc).logits.squeeze(-1).tolist()   # higher = more relevant

The promotion/gate logic (which item to move to slot 1, and when) is not in the model — see the base model card for the decision-logic snippet. This repo only provides the CPU-optimized scorer. Trained on synthetic data; validate on your own before production. See the base model card for full evaluation, limitations, and license/attribution.