OpenMed

OpenMed mmBERT ClinicalScope Spanish Base v1

OpenMed on Hugging Face OpenMed SDK on GitHub pip install openmed License Apache-2.0

OpenMed's open model for negation and uncertainty in Spanish clinical text. Mark one word of a Spanish clinical sentence and it says whether the word is inside a negation scope (no se observa neumonía), inside an uncertainty scope (probable infección urinaria) or outside both. Trained and evaluated by OpenMed. Apache-2.0, 307.5M parameters, runs on CPU.

Developed by OpenMed
Task Negation and uncertainty scope of one marked word, three classes: negation_scope, uncertainty_scope, outside_scope
Input One Spanish sentence with one word marked: Texto: ...⟦word⟧...
Output the class and its score
Parameters 307.5M
Decoding Highest-scoring class, no threshold
Language Spanish
License Apache-2.0

About OpenMed

OpenMed builds open models and open-source tools for clinical and biomedical text that run on your own hardware.

Quick start

pip install -U "transformers>=5.17" torch
import re

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

repo = "OpenMed/mmBERT-ClinicalScope-Spanish-Base-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo).eval()


def mark(sentence, start, end):
    """The evaluated input: "Texto: " and the sentence, with the word between ⟦ and ⟧."""
    return "Texto: " + sentence[:start] + "⟦" + sentence[start:end] + "⟧" + sentence[end:]


def scope(sentence, words=None):
    """Label every word of the sentence (or only `words`): negation, uncertainty or outside scope."""
    found = [m for m in re.finditer(r"\w+(?:-\w+)*", sentence) if words is None or m.group() in words]
    texts = [mark(sentence, m.start(), m.end()) for m in found]
    with torch.inference_mode():
        probs = model(**tokenizer(texts, padding=True, return_tensors="pt")).logits.softmax(-1)
    return [(m.group(), model.config.id2label[int(p.argmax())], float(p.max())) for m, p in zip(found, probs)]


sentence = "No se observa neumonía ni derrame pleural; probable infección urinaria."
for word, label, score in scope(sentence):
    print(f"{word:<10} {label:<17} {score:.2f}")
No         outside_scope     1.00
se         negation_scope    0.92
observa    negation_scope    1.00
neumonía   negation_scope    1.00
ni         negation_scope    0.99
derrame    negation_scope    1.00
pleural    negation_scope    1.00
probable   uncertainty_scope 0.73
infección  uncertainty_scope 1.00
urinaria   uncertainty_scope 0.97

How it reads the input

  • mark() builds the input the model was trained and evaluated on: Texto: and the sentence, with the marked word between ⟦ and ⟧. Keep it exactly as written and mark one word at a time.
  • scope() marks every word of the sentence in turn, or only the words you pass, for example scope(sentence, words={"neumonía", "infección"}). Pass one sentence at a time.
  • The label is the highest-scoring of the three classes, the decoding behind the evaluation; there is no threshold. Scores are not calibrated probabilities.
  • Read the labels of nouns and adjectives: those are the words it was trained and scored on. Cue words (no, sin, probable) and verbs also get a label, but those were never scored.
  • The training sentences had their punctuation split off by spaces (neumonía ;). Normal punctuation works: the quick-start sentence written that way gives the same labels.

Many sentences, GPU

  • Many sentences: scope() batches the words of one sentence; call it in a loop for many sentences.
  • On a GPU, move the model with model.to("cuda") and the batch with tokenizer(...).to("cuda").

Evaluation

Held-out Spanish clinical sentences: 2,028 marked words (867 inside a negation scope, 165 inside an uncertainty scope, 996 outside both) in 1,277 sentences of an existing human-annotated corpus of Spanish clinical text with negation and uncertainty scopes. The held-out sentences come from two clinical specialties (cardiology and traumatology) that were kept out of training and tuning. The model was trained on 7,755 marked words from 14 other specialties for two fixed epochs, checked on 2,258 more from two further specialties (internal medicine and urology), and these 2,028 were scored once at the end, with the model, the references and the scoring code frozen first. Marked words are nouns and adjectives of three or more letters; the cue words themselves (no, sin, posible) are never marked. The source has no patient identifiers, so the split is by specialty and sentence, not by patient. Macro F1 averages the F1 of the three classes.

Model Macro F1 Accuracy
OpenMed mmBERT ClinicalScope Spanish Base v1 (this model) 93.42 95.86
Supervised lexical reference: word and character TF-IDF with logistic regression, trained on the same marked words with the marked sentence and six words of context on each side 80.40 86.29
fastino/GLiNER2.5-Decide-1B zero-shot with the same task and prompts 33.26 41.52
Before OpenMed fine-tuning: jhu-clsp/mmBERT-base with a new, untrained three-class head (a control, not a baseline) 23.95 24.46
Lexical control that sees only the marked word, no context 49.52 59.71

This model scores +13.02 macro F1 points above the supervised lexical reference (95% CI +10.51 to +15.49, paired bootstrap over 1,274 sentence groups, 2,000 resamples). Against the untrained head the change is +69.47 points, but that head is only a control: the real hurdles are the lexical reference and the Decide zero-shot row (+60.16 points, 95% CI +57.96 to +62.22). The lexical control that sees only the marked word reaches 49.52: scope depends on the context, not on the word.

By class

Class Marked words Precision Recall F1 F1, lexical reference
negation_scope 867 96.12 97.23 96.67 87.81
uncertainty_scope 165 93.10 81.82 87.10 65.49
outside_scope 996 96.02 96.99 96.50 87.90

Uncertainty is the hardest class: 87.10 F1, recall 81.82. Of the 165 uncertain words it finds 135; it reads 18 as outside and 12 as negation. Uncertain words are also the rarest in the data.

Two harder slices: in sentences that hold both a negation and an uncertainty scope (107 marked words), accuracy is 91.59 (lexical reference 63.55); for words outside scope in sentences that do have a scope (705), recall is 96.17 (lexical reference 85.67), so the model does not stretch a scope over the whole sentence.

The OpenMed Spanish clinical scope models

Same training words, same 2,028 held-out words, same lexical reference (80.40 macro F1). The mmBERT model scores higher overall at about a quarter of the parameters; the Decide model finds 2 more of the 165 uncertain words.

Model Parameters Code Macro F1 Uncertainty F1 Change vs lexical reference (95% CI)
OpenMed mmBERT ClinicalScope Spanish Base v1 (this model) 307.5M Transformers 93.42 87.10 +13.02 (+10.51 to +15.49)
OpenMed GLiNER2.5 ClinicalScope Spanish Decide 1B v1 1.19B OpenMed SDK 91.83 84.83 +11.43 (+9.06 to +13.72)

Examples

Real outputs of this model with the quick-start scope() helper (Transformers 5.22.1, on CPU). The sentences are synthetic, written in Spanish for this card. Each word was marked in turn; runs of words inside a scope are in bold with their scope, and words outside both are plain.

Quick start: No se observa neumonía ni derrame pleuralnegation; probable infección urinariauncertainty.

  1. Negation list: Sin fiebre ni disneanegation; refiere dolor torácico opresivo desde ayer.
  2. Probable: Probable infección urinariauncertainty, pendiente de urocultivo.
  3. Ruled out: Se descarta trombosis venosa profunda mediantenegation ecografía doppler.
  4. Suspicion: Sospecha de fractura de escafoidesuncertainty, se solicita resonancia magnética.
  5. Not ruled out: No puede descartarsenegation una rotura parcial del ligamento cruzado anterioruncertainty.
  6. Negation then a finding: No se palpan adenopatías cervicalesnegation y el abdomen es blando y depresible.
  7. Both scopes: Niega dolor abdominalnegation, aunque podría tratarse de una apendicitis incipienteuncertainty.
  8. Double negation: No es imposiblenegation que exista una lesión meniscaluncertainty.
  9. Scope to the end: No hay evidencia de metástasis hepáticas ni óseas en el estudio actualnegation.
  10. Hedged impression: Impresiona de celulitis, sin datos de abscesonegation.
  11. Plan to rule out: Descartar tromboembolismo pulmonarnegation.
  12. Family history: Su padre falleció de un infarto de miocardio.

Good to know

  • A scope stops at the end of its clause: in example 1, fiebre and disnea are negated and dolor torácico after the semicolon is outside; in the quick start the negation stops at the semicolon and probable opens an uncertainty scope.
  • One sentence can hold both scopes: example 7 negates dolor abdominal and puts apendicitis incipiente in an uncertainty scope.
  • A negated doubt reads as uncertainty, not negation: No puede descartarse una rotura (example 5) and the double negation No es imposible que exista una lesión (example 8) put rotura and lesión in an uncertainty scope.
  • It misses some hedges: Impresiona de celulitis (example 10) leaves celulitis outside, although the phrase means the finding is only suspected. Sospecha de (example 4) is caught.
  • A plan reads as a negation: Descartar tromboembolismo pulmonar (example 11) puts tromboembolismo in a negation scope, though nothing has been ruled out yet.
  • A negation scope can run to the end of the sentence: estudio actual, where the findings were looked for, is inside the negation in example 9. In example 3 the scope takes in mediante and stops before ecografía doppler.
  • Outside is not the same as present: infarto in example 12 is outside both scopes, but it is the father's history. The model reads negation and uncertainty only; it does not say who has a condition, or when.
  • Most scores sit near 1.00 (77 of the 113 words above round to 1.00), so use them to rank words, not as probabilities.
  • Cue words mostly come back outside (No in the quick start, Sin and Sospecha in examples 1 and 4). They were never scored, so read the findings.

Limitations

  • It labels one marked word at a time for negation and uncertainty scope. It is not a full clinical assertion model: outside does not mean a condition is present, and it does not handle family history, time, hypotheticals or who the condition belongs to.
  • Trained and tested on held-out sentences of one Spanish clinical corpus, split by specialty and sentence, not by patient; patient-level independence cannot be checked. It has not been validated on other institutions, other kinds of notes or other languages; the examples above are synthetic.
  • Uncertainty is the weak class: 87.10 F1 and 81.82 recall; it missed 30 of the 165 uncertain words in the evaluation.
  • Only nouns and adjectives were marked in training and evaluation. Labels on cue words, verbs and function words were not scored.
  • Scores are not calibrated probabilities.
  • It labels words. It does not give medical advice and is not a medical device.

Built on

OpenMed fine-tuned this model from jhu-clsp/mmBERT-base by Johns Hopkins University CLSP (MIT), built with the open-source Transformers library. Training and evaluation data: held-out and training sentences of an existing human-annotated corpus of Spanish clinical text, annotated for negation and uncertainty cues and their scopes. Thank you to Johns Hopkins University CLSP for releasing mmBERT-base openly.

This model is released under Apache-2.0. No training or evaluation text is included in this repository.

Citation

@misc{panahi2025openmedner,
  title={OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets},
  author={Maziyar Panahi},
  year={2025},
  eprint={2508.01630},
  archivePrefix={arXiv}
}

@misc{marone2025mmbert,
  title={mmBERT: A Modern Multilingual Encoder with Annealed Language Learning},
  author={Marc Marone and Orion Weller and William Fleshman and Eugene Yang and Dawn Lawrie and Benjamin Van Durme},
  year={2025},
  eprint={2509.06888},
  archivePrefix={arXiv}
}

OpenMed · Hugging Face · GitHub · PyPI · X

Downloads last month
25
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for OpenMed/mmBERT-ClinicalScope-Spanish-Base-v1

Finetuned
(218)
this model

Papers for OpenMed/mmBERT-ClinicalScope-Spanish-Base-v1

Article mentioning OpenMed/mmBERT-ClinicalScope-Spanish-Base-v1

Evaluation results

  • Macro F1 (three classes) on Held-out Spanish clinical sentences (2,028 marked words)
    self-reported
    93.420
  • Accuracy on Held-out Spanish clinical sentences (2,028 marked words)
    self-reported
    95.860
  • F1 negation_scope on Held-out Spanish clinical sentences (2,028 marked words)
    self-reported
    96.670
  • F1 uncertainty_scope on Held-out Spanish clinical sentences (2,028 marked words)
    self-reported
    87.100
  • F1 outside_scope on Held-out Spanish clinical sentences (2,028 marked words)
    self-reported
    96.500