| | --- |
| | language: en |
| | license: apache-2.0 |
| | library_name: transformers |
| | base_model: jhu-clsp/ettin-encoder-17m |
| | model_name: cross-encoder-ettin-17m-DistillRankNET |
| | source: https://github.com/xpmir/cross-encoders |
| | paper: http://arxiv.org/abs/2603.03010 |
| | tags: |
| | - cross-encoder |
| | - sequence-classification |
| | - tensorboard |
| | datasets: |
| | - msmarco |
| | pipeline_tag: text-classification |
| | --- |
| | |
| | # cross-encoder-ettin-17m-DistillRankNET |
| |
|
| | [](http://arxiv.org/abs/2603.03010) |
| | [](https://huggingface.co/collections/xpmir/reproducing-cross-encoders) |
| | [](https://github.com/xpmir/cross-encoders) |
| |
|
| | This model is a cross-encoder based on `jhu-clsp/ettin-encoder-17m`. It was trained on Ms-Marco using loss `distillRankNET` as part of a reproducibility paper for training cross encoders: "**[Reproducing and Comparing Distillation Techniques for Cross-Encoders](http://arxiv.org/abs/2603.03010)**", see the paper for more details. |
| |
|
| |
|
| | ### Contents |
| | - [Model Description](#model-description) |
| | - [Usage](#usage) |
| | - [Evals](#evaluations) |
| |
|
| |
|
| | ## Model Description |
| |
|
| | This model is intended for **re-ranking** the top results returned by a retrieval system (like BM25, Bi-Encoders or SPLADE). |
| |
|
| | - **Training Data:** MS MARCO Passage |
| | - **Language:** English |
| | - **Loss** distillRankNET |
| |
|
| | Training can be easily reproduced using the assiciated repository. |
| | The exact training configuration used for this model is also detailed in [config.yaml](./config.yaml). |
| |
|
| | ## Usage |
| |
|
| | Quick Start: |
| | ```python |
| | from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| | import torch |
| | |
| | tokenizer = AutoTokenizer.from_pretrained("xpmir/cross-encoder-ettin-17m-DistillRankNET") |
| | model = AutoModelForSequenceClassification.from_pretrained("xpmir/cross-encoder-ettin-17m-DistillRankNET") |
| | |
| | features = tokenizer("What is experimaestro ?", "Experimaestro is a powerful framework for ML experiments management...", padding=True, truncation=True, return_tensors="pt") |
| | |
| | model.eval() |
| | with torch.no_grad(): |
| | scores = model(**features).logits |
| | print(scores) |
| | ``` |
| |
|
| | ## Evaluations |
| |
|
| | We provide evaluations of this cross-encoder re-ranking the top `1000` documents retrieved by `naver/splade-v3-distilbert`. |
| |
|
| | | dataset | RR@10 | nDCG@10 | |
| | |:-------------------|:----------|:----------| |
| | | msmarco_dev | 23.94 | 28.98 | |
| | | trec2019 | 80.25 | 53.28 | |
| | | trec2020 | 86.96 | 53.33 | |
| | | fever | 62.54 | 64.01 | |
| | | arguana | 8.17 | 12.20 | |
| | | climate_fever | 14.74 | 10.74 | |
| | | dbpedia | 56.75 | 30.47 | |
| | | fiqa | 25.77 | 20.02 | |
| | | hotpotqa | 54.49 | 38.89 | |
| | | nfcorpus | 44.88 | 24.70 | |
| | | nq | 30.91 | 35.10 | |
| | | quora | 72.67 | 73.54 | |
| | | scidocs | 14.23 | 7.60 | |
| | | scifact | 44.12 | 46.84 | |
| | | touche | 57.82 | 30.06 | |
| | | trec_covid | 78.54 | 57.16 | |
| | | robust04 | 51.90 | 30.83 | |
| | | lotte_writing | 51.33 | 41.45 | |
| | | lotte_recreation | 43.58 | 38.96 | |
| | | lotte_science | 33.19 | 27.43 | |
| | | lotte_technology | 32.42 | 25.31 | |
| | | lotte_lifestyle | 53.95 | 45.13 | |
| | | **Mean In Domain** | **63.72** | **45.20** | |
| | | **BEIR 13** | **43.51** | **34.72** | |
| | | **LoTTE (OOD)** | **44.39** | **34.85** | |
