D2C SapModernBERT-XL

SapModernBERT-XL is the biomedical concept retrieval encoder made available to students in UAB's INFO 662/762 course for teaching purposes. It fine-tunes BioClinical-ModernBERT-base with SapBERT-style synonym self-alignment, mapping mention surface strings and concept names into a shared, 768-dimensional embedding space.

The retrieval contract is CLS pooling, L2 normalization, a maximum input length of 32 tokens, and no mention markers.

Intended use

Use the encoder to retrieve biomedical concepts from a vocabulary whose names have been embedded with the same checkpoint, tokenizer, pooling, normalization, and truncation settings. It supports terminology matching, candidate generation, and biomedical entity-linking research.

Usage

Use PyTorch and Transformers 5.5.4, the version tested with this checkpoint and its v5 tokenizer export. The example uses the release repository ID; a local checkpoint directory can be supplied instead. Pin the immutable Hub commit SHA when using this model in the course.

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "UAB-NLP/sapmodernbert-xl"
device = "cuda" if torch.cuda.is_available() else "cpu"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(
    model_id, attn_implementation="sdpa"
).to(device).eval()

@torch.inference_mode()
def embed(names):
    inputs = tokenizer(
        names, padding=True, truncation=True, max_length=32,
        return_tensors="pt",
    ).to(device)
    cls = model(**inputs).last_hidden_state[:, 0, :]
    return F.normalize(cls.float(), p=2, dim=-1)

queries = embed(["heart attack"])
candidates = embed(["myocardial infarction", "asthma"])
scores = queries @ candidates.T  # cosine similarity, shape [1, 2]
print(scores.cpu())

Training data

Alignment sources

The alignment corpus was assembled from UMLS 2026AA English names and synonyms and handwritten modifier cues. UMLS atom selection used LAT=ENG, nonempty STR, and SUPPRESS != 'O'.

Source	Role in the corpus
UMLS DISO, CHEM, and ANAT concept tables	Selected concepts from the project's disorder, chemical/drug, and anatomy tables.
UMLS MRSTY procedure/laboratory selection	Added concepts assigned T058 (Health Care Activity), T059 (Laboratory Procedure), T060 (Diagnostic Procedure), T061 (Therapeutic or Preventive Procedure), or T034 (Laboratory or Test Result).
MedMentions full	Selected 34,724 distinct annotated CUIs before UMLS synonym availability filtering. The builder reads full/data/corpus_pubtator.txt, not st21pv.
HPO in UMLS	Included eligible atoms with source vocabulary SAB=HPO, including concepts outside the preceding selections. This is the ontology source, not the HPO GSC+ annotated corpus.
CLCO	Added 84 clinical-context ontology entries.
Modifier-cue lexicons	Added six pseudo-concepts: negation, uncertainty, severity, course, conditionality, and subject.

These selections overlap. For preselected CUIs, eligible synonyms were collected across UMLS source vocabularies; HPO-only additions outside that set contributed HPO atoms. The model was not restricted to MeSH or SNOMED CT names.

Corpus size and effective training signal

The retained union contains 2,029,747 entries. Pair construction strips whitespace, deduplicates synonyms case-insensitively, skips entries with fewer than two distinct names, and samples at most 15 unordered positive pairs per class.

Entry type	Assembled entries	Classes contributing pairs	Positive pairs
UMLS concepts	2,029,657	1,211,809	4,648,745
CLCO entries	84	0	0
Modifier pseudo-concepts	6	6	90
Total	2,029,747	1,211,815	4,648,835

Semantic coverage

Joining pair-contributing UMLS CUIs to UMLS 2026AA MRSTY identifies 126 semantic types across all 15 official semantic groups. This is representation, not exhaustive coverage of every concept in those groups. Coverage is concentrated in the primary selections above.

The table uses the official NLM semantic-group mapping. In particular, T034 belongs to PHEN, although the project's routing map includes it under DISO. The four explicitly selected procedure types are a subset of the seven official PROC types observed in the effective training set. A concept may have multiple semantic types.

Group	Name	Types	TUI identifiers
ACTI	Activities & Behaviors	9	T051, T052, T053, T054, T055, T056, T057, T064, T066
ANAT	Anatomy	11	T017, T018, T021, T022, T023, T024, T025, T026, T029, T030, T031
CHEM	Chemicals & Drugs	20	T103, T104, T109, T114, T116, T120, T121, T122, T123, T125, T126, T127, T129, T130, T131, T192, T195, T196, T197, T200
CONC	Concepts & Ideas	12	T077, T078, T079, T080, T081, T082, T089, T102, T169, T170, T171, T185
DEVI	Devices	3	T074, T075, T203
DISO	Disorders	12	T019, T020, T033, T037, T046, T047, T048, T049, T050, T184, T190, T191
GENE	Genes & Molecular Sequences	4	T028, T085, T086, T087
GEOG	Geographic Areas	1	T083
LIVB	Living Beings	21	T001, T002, T004, T005, T007, T008, T010, T011, T012, T013, T014, T015, T016, T096, T097, T098, T099, T100, T101, T194, T204
OBJC	Objects	5	T071, T072, T073, T167, T168
OCCU	Occupations	2	T090, T091
ORGA	Organizations	4	T092, T093, T094, T095
PHEN	Phenomena	6	T034, T038, T067, T068, T069, T070
PHYS	Physiology	9	T032, T039, T040, T041, T042, T043, T044, T045, T201
PROC	Procedures	7	T058, T059, T060, T061, T062, T063, T065

MedMentions split exposure

The builder reads the full MedMentions annotation file without a train/development/test filter. Concept identities from held-out splits could therefore affect vocabulary selection, although their annotated mention strings and contexts were not used as alignment inputs. MedMentions results should not be presented as evaluation with an unseen concept inventory.

Training procedure

The retained trainer implements a SapBERT-style, rather than a byte-for-byte reproduction of the original SapBERT, objective. It uses L2-normalized CLS embeddings, within-class positive pairs, in-batch other-class negatives, and Multi-Similarity loss with hard-pair mining.

Setting	Recorded value
Initialization	BioClinical-ModernBERT-base
Positive pair list	4,648,835 pairs from 1,211,815 classes
Pair cap	15 per class
Batch size	128 pairs, corresponding to 256 encoded strings in a full batch
Epochs	3
Output checkpoint	sapmodernbert_xl_final

The retained implementation fixes the MS-loss parameters at alpha=1, beta=60, base=0.5, uses mining epsilon 0.1, and uses AdamW with zero weight decay. Pair sampling uses Python random seed 33; epoch shuffling uses 33 + the zero-based epoch number.

Evaluation

The completed record contains an intrinsic synonym-retrieval sanity check over 300 queries and 2,300 candidate names:

Checkpoint	Accuracy@1	Accuracy@5
Before alignment	0.0933	0.1300
After alignment	0.9100	0.9667

This check samples names from the alignment corpus. The implementation does not remove those concepts or pairs from training, so these scores are not held-out benchmark performance. They measure alignment behavior on a small corpus-derived sample, not end-to-end clinical entity-linking accuracy. Downstream performance also depends on vocabulary coverage, mention detection, candidate retrieval, and reranking; scores from those complete systems should not be attributed to the encoder alone.

Limitations

• Alignment is English-only and reflects the source vocabularies and concept-selection recipe.
• Surface strings can be ambiguous. Context-sensitive distinctions, including modifier meanings, are not established by clustering names or grouping cue words into six broad classes.

License and release contents

Model weights are released under MIT, consistent with the base checkpoint's stated license. The upstream MIT copyright and permission notice is preserved in LICENSE. The custom synonym alignment was developed by the UAB NLP Diffuse2Concept project. The encoder release consists of weights, tokenizer, configuration, license, and documentation. UMLS source files, synonym-pair dumps, concept catalogs, and FAISS indices are separate artifacts and are not included. The model license does not replace the terms applicable to external terminology resources used to construct a candidate vocabulary.

Artifact	SHA-256
model.safetensors	d0650da1c3be70fc165d331f1a6fe58dbb4884a693d0b6553929b7cacd9c161a
align_union_concepts.jsonl	ec305840aa0031bf15e2626daa07d8ded54e55038b6c737d84a8e14e0ed95452

References

• BioClinical-ModernBERT-base: upstream model and pretraining details.
• MedMentions: source corpus and full/st21pv variants.