The evidence toolkit: one patient's question through six open OpenMed models
A woman about to start docetaxel for early breast cancer has one question for her oncology nurse before the first infusion:
Will a cold cap stop my hair from falling out during chemo?
The nurse needs two answers. One she can give today, in plain words. And one she can stand behind: what the trials found, how the trial was run, which procedures it actually tested, and whether the number she is about to quote is really in the trial report.
That is a small evidence review, and it has five steps. We gave each step to an open OpenMed model released this week, six models in all, and ran the whole review on CPU. The patient, her question and every text we fed the models are synthetic, written for this article: twelve patient-information answers, twelve literature snippets, one abstract and one trial report. The abstract and the report describe an invented trial, and their numbers are made up. The outputs are real: each model ran its own model card's quick-start code, with only the inputs swapped.
Each step below has three parts: what the model did in this run, the held-out number that says how far to trust it, and where it went wrong. Five questions with twelve candidates each prove little on their own; the held-out tests are where the numbers come from.
Step 1. Find the answer to give the patient
Model: EmbeddingGemma2 BiomedicalEvidence 270M. It turns a health question and a pool of patient-information answers into vectors, so the question lands closest to the answer that answers it.
The pool held twelve answers, most of them about chemotherapy and several about hair: scalp cooling, wigs, hair care, regrowth, and two other cooling treatments (frozen gloves for numb fingers, ice chips for mouth sores). The model ranked the scalp cooling answer first, at 0.715.
Only just. The wig answer scored 0.712 and the hair-care answer 0.706. All three are about hair during chemotherapy, and only one answers the question. Four more patient questions went through the same pool, on hair regrowth, numb fingers, radiation to the breast and washing hair, and each one got the answer written for it first. The base model, Google's EmbeddingGemma 2, also ranked all five first.
Twelve answers make an easy test. The held-out one is harder: 1,966 questions from nine MedQuAD collections of U.S. government health websites, each searching all 15,139 answers. There, OpenMed's fine-tuning raised NDCG@10 from 81.64 to 91.76, +10.12 points (95% CI +8.87 to +11.20). The base model already found most right answers somewhere in its top ten (Recall@10 93.03, now 95.79). What the fine-tuning changed is the top of the list: the first answer was a right one for 82.95% of questions, up from 65.18%.
Where it goes wrong. Scores are not probabilities and there is no fixed cutoff. The right answer at 0.715 and a neighbor that does not answer the question at 0.712 is what a pool of answers on one topic looks like, so show a person the top few answers, not only the first. On the held-out test, Recall@10 fell 0.59 points in one of the nine collections (GARD) while NDCG@10 rose in all nine.
Step 2. Find the literature
Model: GTE-ModernBERT Biomedical Embedding Base. The same idea for research questions and PubMed-style passages, with no prefixes.
The nurse's literature question: Does scalp cooling reduce chemotherapy-induced alopecia in women with breast cancer? Against twelve snippets, the model put all three scalp cooling snippets on top: the trial result first (0.851), then the safety snippet and the regimen snippet. The base model put a snippet about how distressing hair loss is in second place.
Five research questions ran in all, and four got their snippet first. The miss: Which chemotherapy regimens make scalp cooling less effective? ranked the general trial snippet first (0.820) and the snippet that answers it, about anthracycline regimens, second (0.819). The base model made the same mistake.
On 200 held-out BioASQ questions, each searching a catalog of 16,970 snippets, OpenMed's fine-tuning raised NDCG@10 from 85.10 to 86.81: +1.71 points (95% CI +0.34 to +2.98). That gain is small. The base model is already strong at this, and the lower end of the interval is near zero, so expect this model to behave much like its base, a little better on biomedical literature.
Where it goes wrong. Relevance in the held-out test is known-positive: only the snippets the BioASQ experts selected count, so other snippets that also answer a question count against it. And load it with the card's code, not SentenceTransformer(...): the repository has no sentence-transformers configuration, so that loader falls back to mean pooling and gives different vectors.
Step 3. Sort the abstract
Models: ModernBERT BiomedicalSections Base and GLiNER2.5 BiomedicalSections Small. Each reads one sentence of an abstract and says background, objective, methods, results or conclusions.
The trial the nurse found has an abstract without headings, seven sentences in one block. Both models sorted six of them where they belong. The seventh is the aim:
We tested whether scalp cooling reduces hair loss in women receiving taxane-based chemotherapy.
Both models read it as background (ModernBERT 0.50, GLiNER2.5 0.66). Rewritten as To test whether scalp cooling reduces hair loss ..., both return objective at 1.00.
That is the weak spot the held-out test found too. On 5,976 sentences from 500 abstracts of randomized controlled trials, macro F1 was 80.30 for ModernBERT and 77.44 for GLiNER2.5, against 72.97 for a TF-IDF and logistic regression reference trained on the same sentences (ModernBERT +7.34 over it, 95% CI +5.88 to +8.82). The objective label is the weakest: 62.12 F1 for ModernBERT, and 175 of its 454 objective sentences came back as background.
Where it goes wrong. Out of context, an aim written as We tested whether ... reads like background, and both models label each sentence on its own wording, with no neighbors and no position. If your abstracts state their aims that way, expect background. GLiNER2.5's objective recall is also lower than its base model's before fine-tuning (50.66 against 66.74), because the base model called most sentences objective. An abstract that already has headings already has its sections.
Step 4. Pull the procedures it tested
Model: GLiNER2.5 ProceduresTests Small. It finds diagnostic, laboratory and therapeutic or preventive procedures in biomedical text, with exact character offsets.
On the same abstract, it found chemotherapy, taxane-based chemotherapy (twice), complete blood counts as a laboratory procedure (0.41) and scalp cooling twice (0.41 and 0.82). It missed scalp cooling in the aim sentence: rerun with the cutoff lowered to show every score, that mention scored 0.39, just under the 0.4 default. It tagged docetaxel, a drug, as a therapeutic procedure (0.72). And it returned the bare word cooling three times, as in no cooling and the cooling group; whether that counts depends on your annotation guide.
The base model, Fastino's GLiNER2.5 Small at its own best threshold, found the mention our model missed but labeled it a diagnostic procedure (0.86) and a laboratory procedure (0.35). On 200 held-out MedMentions abstracts with 1,017 annotated procedure mentions, OpenMed's fine-tuning raised exact-span F1 from 12.57 to 48.02, +35.45 points (95% CI +31.33 to +39.62).
Where it goes wrong. Laboratory procedures are its weak label: 28.91 F1 and 25.93 recall, so it finds about one laboratory procedure mention in four. Complete blood counts came back at 0.41, just above the cutoff. Scores also move with the text around a mention: run on the two methods sentences alone, the same call returned nothing at all. Pass whole abstracts, as the card says; the evaluation used windows of up to 600 characters.
Step 5. Check the claim against the trial report
Model: DeBERTa-v3 ClinicalTrials NLI Large. Give it one section of a trial report and a statement: it answers entailment (the report supports it), contradiction, or neutral (the report does not settle it).
The abstract's conclusion is that scalp cooling reduced hair loss. Against the trial's results section, the model accepted Hair was preserved in more participants with scalp cooling than without it at 1.00. It also handled a ratio (more than ten times as frequent, entailment 0.91), a negation over the whole trial (No participant in the primary trial reported a headache., contradiction, since six in the cooling arm did) and a statement the report cannot settle (neutral).
Two of the seven statements fooled it, both at a score of 1.00:
- Most participants in cohort 1 of the primary trial kept their hair. The report says 47.6%, which is not most. The model said entailment.
- More than 200 participants were analyzed in the primary trial. The two arms add up to 143 + 71 = 214. The model said contradiction.
On 196 expert-labeled statement and trial-report pairs from 98 breast-cancer trials, none of them in training, OpenMed's fine-tuning raised macro F1 from 62.71 to 83.65, +20.94 points (95% CI +14.08 to +28.44). Part of that gain comes from the base model answering neutral on 21.94% of these pairs; with both models forced to choose entailment or contradiction, the gain is +13.84 points.
Where it goes wrong. About one contradicted statement in five still passes as supported: 20 of 98 (20.41%, down from 28.57%). The scores do not tell a right answer from a wrong one: of the 187 held-out answers it scored 0.90 or more, 30 were wrong. Neutral is unmeasured, because every held-out pair is either supported or contradicted. Sums across arms and statements comparing two trials are where it fails in the card's examples, and it was trained and tested on breast-cancer trial reports only.
What the nurse gets
An answer to read with the patient, ranked first but only just. The trial snippet. The abstract in five labeled sections, with the aim needing a second look. The procedures the trial tested, with one drug to strike out and one mention to add back. And the abstract's main claim, checked against the report and supported. Every step needed a person at the end of it, and every step showed that person where to look.
The six models
| Step | Model | Base model | Parameters | Held-out result |
|---|---|---|---|---|
| 1. Patient answers | EmbeddingGemma2 BiomedicalEvidence 270M v2 | google/embeddinggemma-2 | 271.0M | NDCG@10 81.64 → 91.76 (MedQuAD, 1,966 questions) |
| 2. Literature | GTE-ModernBERT Biomedical Embedding Base v1 | Alibaba-NLP/gte-modernbert-base | 149.0M | NDCG@10 85.10 → 86.81 (BioASQ, 200 questions) |
| 3. Sections | ModernBERT BiomedicalSections Base v1 | answerdotai/ModernBERT-base | 149.6M | macro F1 80.30 (TF-IDF reference 72.97) |
| 3. Sections | GLiNER2.5 BiomedicalSections Small v1 | fastino/gliner2.5-small-v1 | 73.9M | macro F1 77.44 |
| 4. Procedures | GLiNER2.5 ProceduresTests Small v1 | fastino/gliner2.5-small-v1 | 73.9M | F1 12.57 → 48.02 (MedMentions, 200 abstracts) |
| 5. Claim check | DeBERTa-v3 ClinicalTrials NLI Large v1 | MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli | 435M | macro F1 62.71 → 83.65 (196 pairs) |
All six are released under Apache-2.0, are in English, and run on CPU. Each held-out test is the model's own, so the numbers are not comparable across rows. They are development results on splits OpenMed held out, not the official test sets, so they are not comparable to published leaderboard results either.
What it is not
- Not a medical device, and not advice. These models rank, label and check text. None of them decides whether scalp cooling suits a patient, whether a patient is eligible for a trial, or whether a finding is correct, current or safe.
- Not a reviewer. The claim checker reads the one section it is given. It does not find the right section, retrieve evidence, weigh a trial's quality or explain its answer.
- Not calibrated. No score here is a probability. Two of the wrong answers in this run scored 1.00.
- Not validated on clinical notes or health records. The retrievers were tested on consumer health answers and PubMed snippets, the section models on abstracts of randomized controlled trials, the procedure model on PubMed abstracts, the claim checker on breast-cancer trial reports.
- Not a pipeline product. This article chains six separate models by hand. Each one has its own card, its own code and its own failure modes.
Run the review yourself
Each block is the model card's quick start with this article's synthetic inputs. The printed output under each block is what it printed when we ran it on CPU.
Step 1, patient answers:
pip install -U "sentence-transformers[image]"
import torch
from sentence_transformers import SentenceTransformer
model = SentenceTransformer(
"OpenMed/EmbeddingGemma2-BiomedicalEvidence-270M-v2",
model_kwargs={"dtype": torch.float32}, # float32 or bfloat16, never float16
)
# The prefixes the model was trained and evaluated with
QUERY = "task: search result | query: "
DOCUMENT = "title: none | text: "
questions = ["Will a cold cap stop my hair from falling out during chemo?"]
answers = [
(
"Scalp cooling, sometimes called a cold cap, chills the scalp before, "
"during and after each chemotherapy infusion. It can reduce hair loss "
"with some chemotherapy drugs, but it does not work for everyone."
),
(
"Some people wear a wig, scarf or hat during chemotherapy. Choosing a "
"wig before treatment starts makes it easier to match your natural "
"hair color and style."
),
(
"During chemotherapy, wash your hair with a mild shampoo, use a soft "
"brush, and avoid hair dye, perms and hot styling tools."
),
(
"Sucking on ice chips during some chemotherapy infusions can help "
"prevent painful mouth sores."
),
(
"Radiation therapy causes hair loss only in the part of the body being "
"treated. Radiation to the breast does not make the hair on your head "
"fall out."
),
]
q = model.encode([QUERY + t for t in questions], normalize_embeddings=True)
a = model.encode([DOCUMENT + t for t in answers], normalize_embeddings=True)
scores = q @ a.T # cosine similarity
for question, row in zip(questions, scores):
print(question)
for j in row.argsort()[::-1][:3]:
print(f" {row[j]:.3f} {answers[j][:60]}...")
Will a cold cap stop my hair from falling out during chemo?
0.715 Scalp cooling, sometimes called a cold cap, chills the scalp...
0.712 Some people wear a wig, scarf or hat during chemotherapy. Ch...
0.706 During chemotherapy, wash your hair with a mild shampoo, use...
Step 2, literature:
pip install -U transformers torch
import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
repo = "OpenMed/GTE-ModernBERT-Biomedical-Embedding-Base-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModel.from_pretrained(repo, dtype=torch.float32).eval()
def embed(texts):
# The evaluated settings: no prefix, 384 tokens, first-token (CLS) pooling, unit length
batch = tokenizer(
texts, padding=True, truncation=True, max_length=384, return_tensors="pt"
)
with torch.no_grad():
cls = model(**batch).last_hidden_state[:, 0]
return F.normalize(cls, p=2, dim=1)
questions = [
"Does scalp cooling reduce chemotherapy-induced alopecia in women with breast cancer?",
"Which chemotherapy regimens make scalp cooling less effective?",
]
passages = [
(
"In a randomized trial of women receiving taxane-based chemotherapy for "
"early breast cancer, scalp cooling was associated with less hair loss "
"than no cooling."
),
(
"Hair preservation with scalp cooling was less frequent in patients "
"receiving anthracycline-based regimens than in those receiving taxanes "
"alone."
),
(
"Scalp metastases were rare among patients with breast cancer who used "
"scalp cooling, and their incidence was similar to that in patients who "
"did not."
),
(
"Frozen gloves reduced docetaxel-induced nail toxicity in a "
"self-controlled study of patients with solid tumors."
),
(
"Topical minoxidil did not prevent chemotherapy-induced alopecia but "
"shortened the time to hair regrowth."
),
]
scores = embed(questions) @ embed(passages).T # cosine similarity
for question, row in zip(questions, scores):
print(question)
for j in row.argsort(descending=True)[:3]:
print(f" {row[j]:.3f} {passages[j][:60]}...")
Does scalp cooling reduce chemotherapy-induced alopecia in women with breast cancer?
0.851 In a randomized trial of women receiving taxane-based chemot...
0.794 Scalp metastases were rare among patients with breast cancer...
0.792 Hair preservation with scalp cooling was less frequent in pa...
Which chemotherapy regimens make scalp cooling less effective?
0.820 In a randomized trial of women receiving taxane-based chemot...
0.819 Hair preservation with scalp cooling was less frequent in pa...
0.760 Scalp metastases were rare among patients with breast cancer...
Step 3, sections (the OpenMed SDK splits the sentences, the transformers pipeline labels them):
pip install -U openmed transformers torch
from openmed.processing.sentences import segment_text
from transformers import pipeline
abstract = (
"Hair loss is one of the most distressing side effects of chemotherapy "
"for early breast cancer. We tested whether scalp cooling reduces hair "
"loss in women receiving taxane-based chemotherapy. In this randomized "
"trial, 214 women scheduled for at least four cycles of docetaxel were "
"assigned in a 2:1 ratio to scalp cooling or no cooling. Hair loss was "
"graded by a blinded nurse from standardized scalp photographs, and "
"complete blood counts were checked before each cycle. After four "
"cycles, 47.6% of women in the cooling group and 4.2% in the control "
"group kept at least half of their hair. Headache and scalp pain were "
"the most common adverse events with cooling. Scalp cooling reduced hair "
"loss in women receiving taxane-based chemotherapy and was well "
"tolerated."
)
classifier = pipeline(
"text-classification",
model="OpenMed/ModernBERT-BiomedicalSections-Base-v1",
device="cpu",
)
sentences = [s.text.strip() for s in segment_text(abstract)]
for sentence, p in zip(sentences, classifier(sentences)):
print(f"{p['label']:<12} {p['score']:.2f} {sentence}")
BACKGROUND 0.76 Hair loss is one of the most distressing side effects of chemotherapy for early breast cancer.
BACKGROUND 0.50 We tested whether scalp cooling reduces hair loss in women receiving taxane-based chemotherapy.
METHODS 0.99 In this randomized trial, 214 women scheduled for at least four cycles of docetaxel were assigned in a 2:1 ratio to scalp cooling or no cooling.
METHODS 1.00 Hair loss was graded by a blinded nurse from standardized scalp photographs, and complete blood counts were checked before each cycle.
RESULTS 1.00 After four cycles, 47.6% of women in the cooling group and 4.2% in the control group kept at least half of their hair.
RESULTS 0.94 Headache and scalp pain were the most common adverse events with cooling.
CONCLUSIONS 1.00 Scalp cooling reduced hair loss in women receiving taxane-based chemotherapy and was well tolerated.
Step 4, procedures, through the OpenMed SDK (Apache-2.0). The settings are the decoding behind the evaluation: the three prompts as written, threshold=0.05 with confidence_threshold=0.4, and max_length=150.
pip install -U "openmed[gliner2]" # OpenMed 3.2 or later
from openmed import OpenMedConfig, analyze_text
text = (
"Hair loss is one of the most distressing side effects of chemotherapy "
"for early breast cancer. We tested whether scalp cooling reduces hair "
"loss in women receiving taxane-based chemotherapy. In this randomized "
"trial, 214 women scheduled for at least four cycles of docetaxel were "
"assigned in a 2:1 ratio to scalp cooling or no cooling. Hair loss was "
"graded by a blinded nurse from standardized scalp photographs, and "
"complete blood counts were checked before each cycle. After four "
"cycles, 47.6% of women in the cooling group and 4.2% in the control "
"group kept at least half of their hair. Headache and scalp pain were "
"the most common adverse events with cooling. Scalp cooling reduced hair "
"loss in women receiving taxane-based chemotherapy and was well "
"tolerated."
)
result = analyze_text(
text,
model_id="OpenMed/GLiNER2.5-ProceduresTests-Small-v1",
labels=[
"diagnostic procedure",
"laboratory procedure",
"therapeutic or preventive procedure",
],
threshold=0.05,
confidence_threshold=0.4,
max_length=150,
config=OpenMedConfig(use_medical_tokenizer=False),
)
for e in result.entities:
print(e.start, e.end, e.text, f"[{e.label}]", f"{e.confidence:.2f}")
57 69 chemotherapy [therapeutic or preventive procedure] 0.86
164 189 taxane-based chemotherapy [therapeutic or preventive procedure] 0.51
265 274 docetaxel [therapeutic or preventive procedure] 0.72
307 320 scalp cooling [therapeutic or preventive procedure] 0.41
327 334 cooling [therapeutic or preventive procedure] 0.52
417 438 complete blood counts [laboratory procedure] 0.41
512 519 cooling [therapeutic or preventive procedure] 0.46
655 662 cooling [therapeutic or preventive procedure] 0.70
664 677 Scalp cooling [therapeutic or preventive procedure] 0.82
715 740 taxane-based chemotherapy [therapeutic or preventive procedure] 0.43
Step 5, claim check, report section first and statement second, with no truncation:
pip install -U transformers torch
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
model_id = "OpenMed/DeBERTa-v3-ClinicalTrials-NLI-Large-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()
# The trial report section (premise) comes first, the statement (hypothesis) second.
report = (
"Primary trial:\n"
"Outcome Measurement:\n"
" Hair Preservation\n"
" Hair loss of grade 0 or 1 (less than 50% hair loss), graded from standardized scalp photographs.\n"
" Time frame: after 4 cycles, about 12 weeks\n"
"Results 1:\n"
" Arm/Group Title: Scalp Cooling\n"
" Overall Number of Participants Analyzed: 143\n"
" Measure Type: Number\n"
" Unit of Measure: percentage of participants 47.6\n"
"Results 2:\n"
" Arm/Group Title: No Scalp Cooling\n"
" Overall Number of Participants Analyzed: 71\n"
" Measure Type: Number\n"
" Unit of Measure: percentage of participants 4.2\n"
)
statements = [
"Hair was preserved in more participants with scalp cooling than without it.",
"Most participants in cohort 1 of the primary trial kept their hair.",
"More than 200 participants were analyzed in the primary trial.",
]
for statement in statements:
inputs = tokenizer(report, statement, return_tensors="pt")
with torch.inference_mode():
probs = model(**inputs).logits.softmax(-1)[0].tolist()
label = model.config.id2label[probs.index(max(probs))]
scores = ", ".join(f"{model.config.id2label[i]} {p:.2f}" for i, p in enumerate(probs))
print(statement)
print(f" {label} ({scores})")
Hair was preserved in more participants with scalp cooling than without it.
entailment (entailment 1.00, neutral 0.00, contradiction 0.00)
Most participants in cohort 1 of the primary trial kept their hair.
entailment (entailment 1.00, neutral 0.00, contradiction 0.00)
More than 200 participants were analyzed in the primary trial.
contradiction (entailment 0.00, neutral 0.00, contradiction 1.00)
The model cards have the settings, more examples and every caveat: EmbeddingGemma2 BiomedicalEvidence, GTE-ModernBERT Biomedical Embedding, ModernBERT BiomedicalSections, GLiNER2.5 BiomedicalSections, GLiNER2.5 ProceduresTests and DeBERTa-v3 ClinicalTrials NLI.
How the held-out numbers were made
Every model was scored once on questions, sentences or pairs it never saw in training, after its checkpoint, thresholds and scoring code were frozen on separate calibration data. Intervals come from paired bootstraps with 2,000 resamples over the unit that keeps related items together.
- Patient answers: 1,966 MedQuAD development questions held out by OpenMed, each searching 15,139 unique answers; source pages, topics and near-duplicate answer groups kept inside one split; bootstrap over 609 source groups. Relevance comes from the source pages' own question and answer links, not from clinician judgments.
- Literature: 200 BioASQ development questions held out from the training release, each searching 16,970 snippets; questions sharing a paper or a snippet kept inside one split; bootstrap over 160 question groups. The official BioASQ test sets were not used.
- Sections: 5,976 sentences from 500 PubMed abstracts of randomized controlled trials, each sentence labeled with the section its abstract was published under; trained on 38,528 sentences from 3,200 other abstracts; bootstrap over the 500 abstracts.
- Procedures: 200 MedMentions abstracts with 1,017 human-annotated mentions of three procedure types; a prediction counts only if its exact span and its label both match; bootstrap over the abstracts.
- Claim check: 196 expert-labeled statement and trial-report pairs from 98 breast-cancer trials, 98 supported and 98 contradicted, with no trial shared with training; bootstrap over 69 groups of linked trials.
Built on open work
These models stand on open models and open data. Thank you to Google DeepMind for EmbeddingGemma 2, to Tongyi Lab at Alibaba Group for gte-modernbert-base, to Answer.AI for ModernBERT, to Fastino for GLiNER2.5 Small and the GLiNER2 library, and to Moritz Laurer for DeBERTa-v3-large-mnli-fever-anli-ling-wanli, built on Microsoft's DeBERTa-v3-large. Thank you to Asma Ben Abacha and Dina Demner-Fushman for MedQuAD (CC BY 4.0), to the BioASQ organizers and experts for the BioASQ question answering data (CC BY 2.5), and to Sunil Mohan and Donghui Li for MedMentions (CC0). The sentence-transformers, transformers and GLiNER2 libraries made every step a few lines of code. This credit does not imply their endorsement.
And the cold cap answer came first, 0.715 to 0.712. That is why the nurse reads the top three.









