re-cep/vitiligo-relation-extractor

Vitiligo Relation Extractor

A biomedical relation extraction model built for the Vitiligo Initiative, an open, non-profit research platform. Given a passage whose chemicals, diseases, genes and variants are already annotated, it classifies the relation between every pair of them into BioRED's eight types. It reads the text PubTator 3.0 does not: full-text paper bodies and abstracts outside PubMed.

Benchmark	Pair F1	Pair + type F1
BioRED test (100 abstracts, expert-annotated)	0.687	0.540
Agreement with PubTator 3.0 (591 held-out vitiligo abstracts)	0.710	0.611

Model details

• Architecture: BiomedBERT encoder with an entity-pair classifier (ATLOP-style, without context pooling). A passage is encoded once and every entity pair in it is scored from that single pass.
• Base model: BiomedBERT (base, uncased, PubMed abstracts and full text)
• Labels: Association, Positive_Correlation, Negative_Correlation, Bind, Cotreatment, Comparison, Drug_Interaction, Conversion, or none
• Entity types: Chemical, Disease, Gene, Variant
• Parameters: 111M · Language: English · License: Apache-2.0
• Code: vitiligo.relations

Uses

Intended: reading relations out of biomedical literature to build a knowledge graph, where each relation keeps the sentence it came from so it can be checked. In the Vitiligo Initiative, its relations become graph edges with lower confidence than curated or registry sources.

Out of scope: clinical decisions, or any use where an unchecked relation is taken as fact. It does not find entities itself; they must be annotated first, for example by PubTator 3.0.

How to use

The model runs through the project's own code, not a transformers class.

pip install "vitiligo[train] @ git+https://github.com/recepsirin/vitiligo-initiative"

from vitiligo.relations.train import load

model, bias, settings = load("re-cep/vitiligo-relation-extractor")

Append @<revision> to the id to pin one version. To reproduce the BioRED test scores:

vitiligo relations evaluate --model re-cep/vitiligo-relation-extractor --split Test

Training

1. Silver: PubTator 3.0's relations in 11,106 abstracts of the Vitiligo Initiative corpus, excluding every BioRED abstract and a fixed 1-in-20 held-out set.
2. Gold: BioRED's 400 training abstracts, starting from the silver model.

The epoch (8 of 10) and the decision threshold were chosen on BioRED's development split (F1 0.554). Per-epoch development scores are in relations.json.

Evaluation

BioRED test split, relations between given entities, micro-averaged as in the BioRED paper. Pair + type precision is 0.559 and recall 0.522.

Training	Dev F1	Test pair F1	Test pair + type F1
DeBERTa-v3-base on BioRED	0.475	0.645	0.451
BiomedBERT on BioRED	0.499	0.642	0.459
BiomedBERT on PubTator silver	0.510	0.635	0.537
BiomedBERT, silver then BioRED (this model)	0.554	0.687	0.540

The BioRED paper's PubMedBERT reaches 0.729 and 0.589 by encoding each pair separately; this model encodes each passage once, which is what makes reading 821,413 candidate pairs in full texts practical.

In use on full texts. Of 120 relations read from full-text bodies and checked by hand against their sentences, 68% were correct. The Vitiligo Initiative therefore loads only relations with probability ≥ 0.8, outside tables and figures, excluding co-administration claims, and accepting "treats" only when the disease is vitiligo. On a fresh hand-checked sample under that rule, 45 of 60 relations (75%) were correct, and 15 of 17 where vitiligo is one of the pair.

Limitations

• About a quarter of the relations it loads in practice are wrong; each should be read with the sentence it came from.
• Table rows, figure captions and lists of drugs are frequent sources of error.
• The silver training data come from one disease's literature; accuracy on other domains is untested beyond BioRED.

Files

File	Contents
model.safetensors	All weights: encoder, entity-type embedding, pair classifier
config.json, tokenizer.json, tokenizer_config.json	BiomedBERT's configuration and tokenizer, unchanged
relations.json	Label order, decision bias, special-token ids, training history

License and attribution

Released under Apache-2.0. Fine-tuned from BiomedBERT (© Microsoft, MIT License). Trained on BioRED, distributed by NCBI as freely available, and on PubTator 3.0 annotations, a work of the US Government.

Citation

• Luo L, Lai PT, Wei CH, Arighi CN, Lu Z. BioRED: a rich biomedical relation extraction dataset. Briefings in Bioinformatics 2022;23(5):bbac282.
• Gu Y, Tinn R, Cheng H, et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare 2021;3(1).
• Wei CH, Allot A, Lai PT, et al. PubTator 3.0: an AI-powered literature resource for unlocking biomedical knowledge. Nucleic Acids Research 2024;52(W1):W540-W546.
• Zhou W, Huang K, Ma T, Huang J. Document-level relation extraction with adaptive thresholding and localized context pooling. AAAI 2021.