File size: 3,593 Bytes
ff8bd76 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 | ---
language: en
license: apache-2.0
library_name: transformers
base_model: microsoft/MiniLM-L12-H384-uncased
model_name: cross-encoder-MiniLM-L12-DistillRankNET
source: https://github.com/xpmir/cross-encoders
paper: http://arxiv.org/abs/2603.03010
tags:
- cross-encoder
- sequence-classification
- tensorboard
datasets:
- msmarco
pipeline_tag: text-classification
---
# cross-encoder-MiniLM-L12-DistillRankNET
[](http://arxiv.org/abs/2603.03010)
[](https://huggingface.co/collections/xpmir/reproducing-cross-encoders)
[](https://github.com/xpmir/cross-encoders)
This model is a cross-encoder based on `microsoft/MiniLM-L12-H384-uncased`. It was trained on Ms-Marco using loss `distillRankNET` as part of a reproducibility paper for training cross encoders: "**[Reproducing and Comparing Distillation Techniques for Cross-Encoders](http://arxiv.org/abs/2603.03010)**", see the paper for more details.
### Contents
- [Model Description](#model-description)
- [Usage](#usage)
- [Evals](#evaluations)
## Model Description
This model is intended for **re-ranking** the top results returned by a retrieval system (like BM25, Bi-Encoders or SPLADE).
- **Training Data:** MS MARCO Passage
- **Language:** English
- **Loss** distillRankNET
Training can be easily reproduced using the assiciated repository.
The exact training configuration used for this model is also detailed in [config.yaml](./config.yaml).
## Usage
Quick Start:
```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch
tokenizer = AutoTokenizer.from_pretrained("microsoft/MiniLM-L12-H384-uncased")
model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-MiniLM-L12-DistillRankNET")
features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt")
model.eval()
with torch.no_grad():
scores = model(**features).logits
print(scores)
```
## Evaluations
We provide evaluations of this cross-encoder re-ranking the top `1000` documents retrieved by `naver/splade-v3-distilbert`.
| dataset | RR@10 | nDCG@10 |
|:-------------------|:----------|:----------|
| msmarco_dev | 37.40 | 43.98 |
| trec2019 | 96.12 | 74.57 |
| trec2020 | 93.83 | 73.48 |
| fever | 81.21 | 80.95 |
| arguana | 18.48 | 27.97 |
| climate_fever | 27.52 | 20.31 |
| dbpedia | 75.81 | 46.06 |
| fiqa | 43.71 | 36.25 |
| hotpotqa | 85.35 | 66.48 |
| nfcorpus | 57.75 | 34.59 |
| nq | 53.19 | 58.21 |
| quora | 76.34 | 78.62 |
| scidocs | 28.06 | 15.79 |
| scifact | 66.12 | 69.34 |
| touche | 64.33 | 34.46 |
| trec_covid | 87.17 | 70.74 |
| robust04 | 75.25 | 52.28 |
| lotte_writing | 66.66 | 58.11 |
| lotte_recreation | 60.60 | 55.12 |
| lotte_science | 46.01 | 38.34 |
| lotte_technology | 53.36 | 44.41 |
| lotte_lifestyle | 71.62 | 61.69 |
| **Mean In Domain** | **75.78** | **64.01** |
| **BEIR 13** | **58.85** | **49.21** |
| **LoTTE (OOD)** | **62.25** | **51.66** | |