OpenMed

OpenMed GTE-ModernBERT Biomedical Embedding Base v1

OpenMed on Hugging Face OpenMed SDK on GitHub pip install openmed License Apache-2.0

OpenMed's open model for finding evidence in the biomedical literature. It turns English research questions and PubMed-style passages into 768-dimensional vectors, so each question lands closest to the passages that answer it. Trained and evaluated by OpenMed. Apache-2.0, 149.0M parameters, runs on CPU.

Developed by OpenMed
Task Biomedical question to literature passage retrieval (dense embeddings)
Output 768-dimensional vector of unit length; the dot product of two vectors is their cosine similarity
Parameters 149.0M
Max input 384 tokens (the length it was trained and evaluated at)
Pooling first token (CLS), then L2 normalization
Prefixes none
Language English
License Apache-2.0

About OpenMed

OpenMed builds open models and open-source tools for clinical and biomedical text that run on your own hardware.

Quick start

pip install -U transformers torch
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

repo = "OpenMed/GTE-ModernBERT-Biomedical-Embedding-Base-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, dtype=torch.float32).eval()

def embed(texts):
    # The evaluated settings: no prefix, 384 tokens, first-token (CLS) pooling, unit length
    batch = tokenizer(
        texts, padding=True, truncation=True, max_length=384, return_tensors="pt"
    )
    with torch.no_grad():
        cls = model(**batch).last_hidden_state[:, 0]
    return F.normalize(cls, p=2, dim=1)

questions = [
    "Which drugs are CFTR modulators?",
    "What type of mutation causes Huntington disease?",
]
passages = [
    (
        "Cystic fibrosis is caused by mutations in the CFTR gene, which "
        "encodes a chloride channel; the F508del variant is found on at "
        "least one allele in most patients."
    ),
    (
        "Ivacaftor, a CFTR potentiator, increases channel open probability "
        "and improved lung function in patients carrying the G551D "
        "mutation."
    ),
    (
        "The CFTR correctors lumacaftor, tezacaftor and elexacaftor improve "
        "the folding and trafficking of F508del-CFTR to the cell surface."
    ),
    (
        "Huntington disease is caused by an expanded CAG trinucleotide "
        "repeat in exon 1 of the HTT gene, and longer repeats are "
        "associated with earlier onset."
    ),
    (
        "Fragile X syndrome results from expansion of a CGG repeat in the "
        "5' untranslated region of FMR1, leading to gene silencing."
    ),
]

scores = embed(questions) @ embed(passages).T  # cosine similarity

for question, row in zip(questions, scores):
    print(question)
    for j in row.argsort(descending=True)[:3]:
        print(f"  {row[j]:.3f}  {passages[j][:60]}...")
Which drugs are CFTR modulators?
  0.675  The CFTR correctors lumacaftor, tezacaftor and elexacaftor i...
  0.573  Ivacaftor, a CFTR potentiator, increases channel open probab...
  0.370  Cystic fibrosis is caused by mutations in the CFTR gene, whi...
What type of mutation causes Huntington disease?
  0.705  Huntington disease is caused by an expanded CAG trinucleotid...
  0.482  Cystic fibrosis is caused by mutations in the CFTR gene, whi...
  0.407  Fragile X syndrome results from expansion of a CGG repeat in...

The settings

  • No prefixes or instructions: questions and passages are embedded as they are, as in training and evaluation.
  • First-token (CLS) pooling, then L2 normalization, the base model's own pooling. This repository has no sentence-transformers configuration, so use the code above rather than SentenceTransformer(...), which would fall back to mean pooling (see Good to know).
  • max_length=384 is the length used in training and evaluation (2 of the evaluation's 17,170 inputs were longer). The base model accepts up to 8,192 tokens; longer inputs were not evaluated after fine-tuning.
  • dtype=torch.float32 loads the stored float32 weights. On CPU the repository's own helper also runs in float32, as here; the evaluation ran on a GPU with bfloat16 autocast.
  • Cosine scores rank passages for one question; they are not probabilities.

Large collections, GPU

  • Embed the passage collection once, in batches (embed(passages[i:i + 64])), and store the vectors; at search time, embed only the question.
  • On a GPU: model = model.to("cuda"), move each batch with batch = batch.to("cuda") inside embed, and wrap the forward pass in torch.autocast("cuda", dtype=torch.bfloat16), the setting the evaluation used.

Evaluation

BioASQ, 200 held-out development questions: English biomedical questions written by experts for the BioASQ challenge, each with the PubMed abstract snippets the experts selected as evidence. OpenMed split the BioASQ training release into 1,467 training questions, 160 calibration questions and these 200 development questions (160 groups), keeping questions that share a paper or a snippet inside one split and leaving out papers used in OpenMed's earlier training and evaluation runs. Each question searches a catalog of 16,970 snippets: the 2,082 snippets selected for these 200 questions plus the 14,888 training snippets. The checkpoint was chosen on the calibration questions, then the model and scoring code were frozen and these questions were scored once. The official BioASQ test sets were not used, so these numbers are not comparable to published BioASQ results.

Known-positive scoring: only the snippets the BioASQ experts selected count as relevant. Every other snippet counts as not relevant, although some may also answer the question; they were never judged. Recall@10 is the share of a question's selected snippets that appear in the top ten; 73 of the 200 questions have more than ten selected snippets, so it cannot reach 100%.

Model NDCG@10 Recall@10 MRR@10
OpenMed GTE-ModernBERT Biomedical Embedding Base v1 (this model) 86.81 73.28 92.26
Before OpenMed fine-tuning: Alibaba-NLP/gte-modernbert-base, same settings 85.10 71.73 91.44

OpenMed's fine-tuning added +1.71 NDCG@10 points (95% CI +0.34 to +2.98, paired bootstrap over the 160 question groups, 2,000 resamples) and +1.55 Recall@10 points. The gain is small. The base model is already strong on this task, and the interval's lower end is near zero: expect this model to behave much like its base, slightly better on biomedical literature search.

By question type

Question type Questions NDCG@10 before fine-tuning NDCG@10 Change
Factoid 71 85.41 86.11 +0.70
List 43 77.06 80.78 +3.72
Summary 41 86.94 89.10 +2.16
Yes/no 45 90.63 91.59 +0.96

All four question types improved. These are point estimates from the same run, with no interval per type.

The OpenMed biomedical retrieval models

Two open OpenMed embedding models for biomedical search: one matches health questions to FAQ-style answers, the other matches research questions to literature passages. Each was scored on its own held-out questions, so their numbers are not comparable with each other.

Model Finds Base model Parameters Dimensions Max tokens Prefixes NDCG@10 before โ†’ after
OpenMed EmbeddingGemma2 BiomedicalEvidence 270M v2 the answer to a medical question (consumer health FAQ) google/embeddinggemma-2 271.0M 768 512 question and answer prefixes 81.64 โ†’ 91.76 (MedQuAD, 1,966 questions)
OpenMed GTE-ModernBERT Biomedical Embedding Base v1 (this model) evidence passages for a biomedical research question Alibaba-NLP/gte-modernbert-base 149.0M 768 384 none 85.10 โ†’ 86.81 (BioASQ, known-positive, 200 questions)

Examples

Real outputs of this model with the quick-start code (same model call, prefixes and scoring), on CPU. The 13 snippets and 8 questions are synthetic, written for this card in the style of PubMed abstract sentences; none is BioASQ or PubMed text. Each question was written for one snippet (two for the list question) and searches all 13. The table gives the top three with cosine scores, the snippet written for the question in bold, and where it ranks for this model and for the base model with the same code.

ID Snippet
pd1 Pembrolizumab is a humanized IgG4 monoclonal antibody that binds the programmed death-1 (PD-1) receptor on T cells and blocks its interaction with PD-L1 and PD-L2, restoring antitumor immune responses.
ctla4 Ipilimumab blocks cytotoxic T-lymphocyte antigen 4 (CTLA-4), enhancing T-cell activation and proliferation.
cftr-gene Cystic fibrosis is caused by mutations in the CFTR gene, which encodes a chloride channel; the F508del variant is found on at least one allele in most patients.
ivacaftor Ivacaftor, a CFTR potentiator, increases channel open probability and improved lung function in patients carrying the G551D mutation.
correctors The CFTR correctors lumacaftor, tezacaftor and elexacaftor improve the folding and trafficking of F508del-CFTR to the cell surface.
sglt2 In randomized trials, dapagliflozin and empagliflozin reduced the risk of hospitalization for heart failure in patients with reduced ejection fraction, with or without type 2 diabetes.
metformin Metformin lowers blood glucose mainly by reducing hepatic gluconeogenesis, an effect partly mediated by activation of AMP-activated protein kinase.
statin Statins inhibit HMG-CoA reductase, the rate-limiting enzyme of cholesterol synthesis, and lower LDL cholesterol.
htt Huntington disease is caused by an expanded CAG trinucleotide repeat in exon 1 of the HTT gene, and longer repeats are associated with earlier onset.
fmr1 Fragile X syndrome results from expansion of a CGG repeat in the 5' untranslated region of FMR1, leading to gene silencing.
vitd In a large randomized trial of community-dwelling adults, daily vitamin D3 supplementation did not lower the incidence of total, nonvertebral or hip fractures compared with placebo.
brca Germline mutations in BRCA1 and BRCA2 substantially increase the lifetime risk of breast and ovarian cancer.
olaparib Olaparib, an inhibitor of poly(ADP-ribose) polymerase (PARP), prolonged progression-free survival as maintenance therapy in women with BRCA-mutated advanced ovarian cancer.
# Type Question Top three (cosine) Rank of the written snippet: this model / base
1 Summary What is the mechanism of action of pembrolizumab? pd1 0.743; ctla4 0.497; olaparib 0.314 1 / 1
2 Factoid Which gene is mutated in cystic fibrosis? cftr-gene 0.780; fmr1 0.466; correctors 0.440 1 / 1
3 List Which drugs are CFTR modulators? correctors 0.675; ivacaftor 0.573; cftr-gene 0.370 1, 2 / 1, 2
4 Yes/no Do SGLT2 inhibitors reduce hospitalization for heart failure? sglt2 0.666; correctors 0.432; ivacaftor 0.347 1 / 1
5 Factoid What type of mutation causes Huntington disease? htt 0.705; cftr-gene 0.482; brca 0.414 1 / 1
6 Summary How does metformin lower blood glucose? metformin 0.801; sglt2 0.387; statin 0.378 1 / 1
7 Yes/no Does vitamin D supplementation prevent fractures in older adults? vitd 0.818; sglt2 0.326; correctors 0.263 1 / 1
8 Factoid Which PARP inhibitor is used for BRCA-mutated ovarian cancer? olaparib 0.711; brca 0.493; pd1 0.391 1 / 1

Good to know

  • All four BioASQ question types work here: factoid, list, yes/no and summary questions each rank their snippet first. The list question (example 3) returns both modulator snippets on top (correctors 0.675, ivacaftor 0.573), above the CFTR gene snippet (0.370).
  • Snippets on the same subject that answer a different question rank lower: in example 8 the BRCA risk snippet comes second (0.493) after olaparib (0.711); in example 5 the fragile X snippet, also about a repeat expansion, ranks fourth.
  • No misses on these 8 examples, and the base model also ranks the written snippets first on all 8. They are easy; the evaluation searches 16,970 snippets, and there the fine-tuning gain is small (+1.71 NDCG@10). Expect this model to behave much like its base.
  • Use the code above, not SentenceTransformer("OpenMed/GTE-ModernBERT-Biomedical-Embedding-Base-v1"): this repository has no sentence-transformers configuration, so sentence-transformers falls back to mean pooling. On the card's texts that gives different vectors (cosine as low as 0.75 with the evaluated CLS vectors).
  • Scores are not probabilities and there is no fixed cutoff: the written snippets here score from 0.573 to 0.818. Compare scores within one question.

Limitations

  • Evaluated on BioASQ only: expert biomedical questions and PubMed abstract snippets in English. It has not been validated on clinical notes, patient questions, full-text articles or other languages; the examples above are synthetic.
  • Known-positive relevance: snippets the experts did not select were never judged, so real precision may be higher or lower than measured.
  • The gain over its base model is small: +1.71 NDCG@10 points on 200 questions, 95% CI +0.34 to +2.98.
  • Trained and evaluated at 384 tokens on snippet-length passages; long passages were not evaluated.
  • Cosine scores rank passages for one question; they are not calibrated probabilities.
  • The base model's pretraining data is not fully public, so whether it saw these BioASQ questions or PubMed snippets is unknown.
  • It ranks passages. It does not check that a finding is correct or current, does not give medical advice and is not a medical device.

Built on

OpenMed fine-tuned this model from Alibaba-NLP/gte-modernbert-base by Tongyi Lab, Alibaba Group (Apache-2.0), itself built on ModernBERT, with the open-source transformers library. Training and evaluation data: the BioASQ question answering training data (Zenodo record 7655130; Krithara, Nentidis, Bougiatiotis and Paliouras, 2023), questions and evidence snippets written and selected by biomedical experts for the BioASQ challenge (Tsatsaronis et al., 2015), released under CC BY 2.5. OpenMed used the training release only (training11b.json) and turned its questions and snippets into retrieval training pairs. Thank you to Tongyi Lab and to the BioASQ organizers and experts for releasing their work openly; this credit does not imply their endorsement.

This model is released under Apache-2.0. No BioASQ text is included in this repository.

Citation

@misc{panahi2025openmedner,
  title={OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets},
  author={Maziyar Panahi},
  year={2025},
  eprint={2508.01630},
  archivePrefix={arXiv}
}

@article{tsatsaronis2015bioasq,
  title={An overview of the {BIOASQ} large-scale biomedical semantic indexing and question answering competition},
  author={Tsatsaronis, George and Balikas, Georgios and Malakasiotis, Prodromos and Partalas, Ioannis and Zschunke, Matthias and Alvers, Michael R. and Weissenborn, Dirk and Krithara, Anastasia and Petridis, Sergios and Polychronopoulos, Dimitris and Almirantis, Yannis and Pavlopoulos, John and Baskiotis, Nicolas and Gallinari, Patrick and Arti{\`e}res, Thierry and Ngonga Ngomo, Axel-Cyrille and Heino, Norman and Gaussier, Eric and Barrio-Alvers, Liliana and Schroeder, Michael and Androutsopoulos, Ion and Paliouras, Georgios},
  journal={BMC Bioinformatics},
  volume={16},
  pages={138},
  year={2015},
  doi={10.1186/s12859-015-0564-6}
}

@article{krithara2023bioasqqa,
  title={{BioASQ-QA}: A manually curated corpus for Biomedical Question Answering},
  author={Krithara, Anastasia and Nentidis, Anastasios and Bougiatiotis, Konstantinos and Paliouras, Georgios},
  journal={Scientific Data},
  volume={10},
  pages={170},
  year={2023},
  doi={10.1038/s41597-023-02068-4}
}

@inproceedings{zhang2024mgte,
  title={mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval},
  author={Zhang, Xin and Zhang, Yanzhao and Long, Dingkun and Xie, Wen and Dai, Ziqi and Tang, Jialong and Lin, Huan and Yang, Baosong and Xie, Pengjun and Huang, Fei and others},
  booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track},
  pages={1393--1412},
  year={2024}
}

OpenMed ยท Hugging Face ยท GitHub ยท PyPI ยท X

Downloads last month
18
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for OpenMed/GTE-ModernBERT-Biomedical-Embedding-Base-v1

Finetuned
(46)
this model

Paper for OpenMed/GTE-ModernBERT-Biomedical-Embedding-Base-v1

Evaluation results

  • NDCG@10 (known-positive) on BioASQ (200 held-out development questions, 16,970 snippets)
    self-reported
    86.810
  • Recall@10 (known-positive) on BioASQ (200 held-out development questions, 16,970 snippets)
    self-reported
    73.280
  • MRR@10 (known-positive) on BioASQ (200 held-out development questions, 16,970 snippets)
    self-reported
    92.260