tmy100000001/LitDD_BERT

LitDD-BERT

The abstract-screening classifier from LitDD, a pipeline that maps PubMed literature to gene–disease entries in the G2P developmental-disorder panel.

Given a title+abstract (TIAB), LitDD-BERT answers one question:

Does this paper present evidence that a gene causes a developmental disorder?

It is the first, high-recall stage of a cascade. Its job is to reduce ~35M PubMed records to a tractable candidate set at very high recall; precision is recovered downstream by a cross-encoder (tmy100000001/LitDD_crossencoder), an LLM adjudication step, and a gene-mention filter. Used alone it is deliberately over-inclusive — see Intended use.

⚠️ Requires transformers >= 4.48

This is a ModernBERT architecture. Earlier versions fail with KeyError: 'modernbert'.

pip install "transformers>=4.48"

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

model_id = "tmy100000001/LitDD_BERT"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

tiab = ("A Novel Homozygous Nonsense HYDIN Gene Mutation p.(Arg951*) in Primary Ciliary "
        "Dyskinesia. We report a consanguineous family in which two affected siblings ...")

enc = tok(tiab, truncation=True, max_length=8192, return_tensors="pt")
with torch.no_grad():
    logits = model(**enc).logits
pred = logits.argmax(-1).item()
print(model.config.id2label[pred])   # RELEVANT / NOT_RELEVANT

Label mapping: 0 = NOT_RELEVANT, 1 = RELEVANT.

The deployed pipeline screens only English records published after 1980; matching that condition will reproduce the numbers below. ModernBERT's context is 8,192 tokens, which no PubMed abstract approaches (observed maximum ~800), so truncation is effectively never triggered.

Intended use

Use it for: first-pass screening of biomedical abstracts for gene→developmental-disorder evidence, at corpus scale.

Do not use it for:

• deciding which disease a paper describes — that is the cross-encoder's job; this model is gene-level only, and a paper about a different disease of the same gene is still positive;
• clinical or diagnostic decision-making;
• a precision-sensitive task on its own. On a random PubMed sample the false-positive rate is 5.8%, so at a realistic <1% prevalence, screen-level precision alone is low by design. This is intentional: the cascade trades screen precision for recall.

Training

Base model	thomas-sounack/BioClinical-ModernBERT-large
Task	binary sequence classification, mean pooling, 2-way head
Training set	17,335 (TIAB, label) pairs — 8,975 positive / 8,360 negative
Composition	clinician-annotated TIABs, plus molecular-mechanism-framed positives added after error analysis, plus literature from independent curated sources (gene-fold disciplined, no leakage into held-out genes)
Hyperparameters	lr 3e-5, weight decay 0.1, 5 epochs, batch 32, seed 42
Selection	5-fold StratifiedGroupKFold on the training portion only; refit on full train; test touched once

~11.7% of training records are titles with no abstract. These are kept deliberately: clinical-genetics titles conventionally state gene, variant and phenotype together, and title-only records are more likely to be positive (34.7%) than the corpus average (25.4%).

Evaluation

This checkpoint (seed 42), on the held-out test set and on independent curated literature whose genes were held out of training:

Metric	Value
Precision	0.900
Recall	0.944
F1	0.921
Held-out external recall (all sources)	98.9%
— premined	99.3%
— HPOA	99.2%
— ClinGen	97.2%
False-positive rate on a random PubMed sample	5.85%

Across three training seeds (42/43/44) the configuration gives F1 0.9227 ± 0.0013 and external recall 98.27 ± 0.57%. The released checkpoint is seed 42, chosen by a pre-registered fixed-seed rule rather than by picking the best run; its metrics sit inside that spread (F1 −1.2 sd) and it has the highest held-out external recall of the three.

Note the false-positive rate on a random PubMed sample varies far more across seeds than F1 does — 5.00 / 5.85 / 10.37% (7.07 ± 2.89%) while F1 varies by 0.0013. At corpus scale that is the difference between roughly 1.7M and 3.5M false positives, and a balanced-ish test set is nearly blind to it. If you are deploying this screen, measure FPR on your own sample rather than inferring it from F1. Those are the figures reported in the paper as the property of the method; the table above is the property of this artefact, and is what you should expect to reproduce by running this checkpoint. Seed-to-seed spread comes from residual training non-determinism, not from data changes.

Comparison under an identical CV→refit→test protocol (same splits, same budget):

Model	F1
LitDD-BERT (this model)	0.965
BioClinical-ModernBERT-large	0.925
BioBERT	0.923
BiomedBERT	0.923
ModernBERT-large	0.911

(Benchmark table measured on the original annotated test set, which is why the LitDD-BERT figure differs from the high-recall operating point in the table above — the two use different training sets and different operating points. Both are reported for transparency.)

Limitations

• English, post-1980 only. The pipeline filters on languages == "eng" and pubdate > 1980 before screening; the model was never trained or evaluated outside that slice.
• Gene-level, not disease-level. It cannot tell you which G2P entry a paper supports.
• Register sensitivity. Error analysis showed the residual misses are papers written in a molecular-mechanism register rather than a case-report register. Augmentation reduced this substantially (held-out-gene recall 84.9% → ~98%) but it is the known failure mode.
• Training and inference both use the model's full 8,192-token context, so abstracts are not truncated in practice (~1% exceed 512 tokens; none exceed 1,024).
• Trained in fp32. An earlier release of this checkpoint was trained in bf16 and scored F1 0.911; the difference is precision, not the seed.
• Trained on developmental-disorder genetics. Behaviour on other disease domains is untested.

Citation

Manuscript under review; citation and DOI to follow. Code: https://github.com/ (repository link to be added at acceptance).

Please also cite the base model, BioClinical ModernBERT.