File size: 3,474 Bytes
267c8a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | ---
pipeline_tag: text-ranking
tags:
- sentence-transformers
- cross-encoder
- reranker
- sentence-similarity
- transformers
base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
language: en
license: apache-2.0
---
# BiomedBERT Reranker
This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext](https://huggingface.co/microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext) using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.
The training dataset was generated using a random sample of [PubMed](https://pubmed.ncbi.nlm.nih.gov/) title-abstract pairs along with similar title pairs.
## Usage (txtai)
This model can be used to score a list of text pairs. This is useful as a reranking pipeline after an initial semantic search operation.
```python
from txtai.pipeline import Similarity
ranker = Similarity(path="neuml/biomedbert-base-reranker", crossencode=True)
ranker("query", ["document1", "document2"])
```
## Usage (Sentence-Transformers)
Alternatively, the model can be loaded with [sentence-transformers](https://www.SBERT.net).
```python
from sentence_transformers import CrossEncoder
model = SentenceTransformer("neuml/biomedbert-base-reranker")
model.predict([["query", "document1"], ["query", "document2"]])
```
## Evaluation Results
Performance of this model is compared to previously released models trained on medical literature.
The following datasets were used to evaluate model performance.
- [PubMed QA](https://huggingface.co/datasets/qiaojin/PubMedQA)
- Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
- [PubMed Subset](https://huggingface.co/datasets/awinml/pubmed_abstract_3_1k)
- Split: test, Pair: (title, text)
- [PubMed Summary](https://huggingface.co/datasets/armanc/scientific_papers)
- Subset: pubmed, Split: validation, Pair: (article, abstract)
Evaluation results are shown below. The [Pearson correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient) is used as the evaluation metric.
| Model | PubMed QA | PubMed Subset | PubMed Summary | Average |
| ----------------------------------------------------- | --------- | ------------- | -------------- | --------- |
| [all-MiniLM-L6-v2](https://hf.co/sentence-transformers/all-MiniLM-L6-v2) | 90.40 | 95.92 | 94.07 | 93.46 |
| [bioclinical-modernbert-base-embeddings](https://hf.co/neuml/bioclinical-modernbert-base-embeddings) | 92.49 | 97.10 | 97.04 | 95.54 |
| [biomedbert-base-colbert](https://hf.co/neuml/biomedbert-base-colbert) | 94.59 | 97.18 | 96.21 | 95.99|
| [**biomedbert-base-reranker**](https://hf.co/neuml/biomedbert-base-reranker) | **97.66** | **99.76** | **98.81** | **98.74** |
| [pubmedbert-base-embeddings](https://hf.co/neuml/pubmedbert-base-embeddings) | 93.27 | 97.00 | 96.58 | 95.62 |
| [pubmedbert-base-embeddings-8M](https://hf.co/neuml/pubmedbert-base-embeddings-8M) | 90.05 | 94.29 | 94.15 | 92.83 |
As expected, this cross-encoder model scores much higher than bi-encoder models and late interaction models. The tradeoff is that this is expensive to run and there is no way to scale it past small batches of data. But it's a great model for re-ranking medical literature.
|