metadata
language: en
license: apache-2.0
library_name: transformers
base_model: jhu-clsp/ettin-encoder-150m
model_name: cross-encoder-ettin-150m-DistillRankNET
source: https://github.com/xpmir/cross-encoders
paper: http://arxiv.org/abs/2603.03010
tags:
- cross-encoder
- sequence-classification
- tensorboard
datasets:
- msmarco
pipeline_tag: text-classification
cross-encoder-ettin-150m-DistillRankNET
This model is a cross-encoder based on jhu-clsp/ettin-encoder-150m. It was trained on Ms-Marco using loss distillRankNET as part of a reproducibility paper for training cross encoders: "Reproducing and Comparing Distillation Techniques for Cross-Encoders", see the paper for more details.
Contents
Model Description
This model is intended for re-ranking the top results returned by a retrieval system (like BM25, Bi-Encoders or SPLADE).
- Training Data: MS MARCO Passage
- Language: English
- Loss distillRankNET
Training can be easily reproduced using the assiciated repository. The exact training configuration used for this model is also detailed in config.yaml.
Usage
Quick Start:
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/ettin-encoder-150m")
model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-150m-DistillRankNET")
features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
model.eval()
with torch.no_grad():
scores = model(**features).logits
print(scores)
Evaluations
We provide evaluations of this cross-encoder re-ranking the top 1000 documents retrieved by naver/splade-v3-distilbert.
| dataset | RR@10 | nDCG@10 |
|---|---|---|
| msmarco_dev | 36.30 | 42.93 |
| trec2019 | 96.98 | 75.59 |
| trec2020 | 93.83 | 72.27 |
| fever | 80.10 | 79.82 |
| arguana | 14.52 | 22.21 |
| climate_fever | 27.00 | 19.76 |
| dbpedia | 75.55 | 45.75 |
| fiqa | 47.54 | 39.63 |
| hotpotqa | 85.28 | 66.73 |
| nfcorpus | 57.92 | 35.41 |
| nq | 53.64 | 58.68 |
| quora | 75.29 | 77.46 |
| scidocs | 28.04 | 15.73 |
| scifact | 68.06 | 70.51 |
| touche | 66.31 | 36.81 |
| trec_covid | 96.50 | 77.81 |
| robust04 | 73.96 | 49.37 |
| lotte_writing | 73.65 | 64.31 |
| lotte_recreation | 62.02 | 56.42 |
| lotte_science | 51.11 | 42.43 |
| lotte_technology | 56.74 | 47.53 |
| lotte_lifestyle | 72.68 | 63.49 |
| Mean In Domain | 75.70 | 63.60 |
| BEIR 13 | 59.67 | 49.72 |
| LoTTE (OOD) | 65.03 | 53.92 |