How to load ONNX version with CrossEncoder class?
#7
by hveigz - opened
How to load the ONNX version directly with the CrossEncoder class?
Hello!
I'm afraid this is not currently possible in Sentence Transformers without some minor hacking, e.g.
from optimum.onnxruntime import ORTModelForSequenceClassification
from sentence_transformers import CrossEncoder
model = CrossEncoder("Alibaba-NLP/gte-reranker-modernbert-base")
model.model = ORTModelForSequenceClassification.from_pretrained("Alibaba-NLP/gte-reranker-modernbert-base")
Note: this is untested.
ONNX and OpenVINO support will be added to Sentence Transformers soon.
- Tom Aarsen
hveigz changed discussion status to closed