File size: 4,760 Bytes
526f371 a9d6e76 526f371 a9d6e76 526f371 a9d6e76 526f371 a9d6e76 526f371 a9d6e76 526f371 a9d6e76 526f371 a9d6e76 526f371 a9d6e76 526f371 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 | ---
license: mit
language: en
tags:
- adverse-drug-events
- drug-safety
- pharmacovigilance
- text-classification
- biomedical
- PubMedBERT
datasets:
- custom
base_model: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract
pipeline_tag: text-classification
library_name: transformers
---
# OnSIDES-BERT: Adverse Drug Event Classifier
A fine-tuned [PubMedBERT](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract) model for classifying whether a medical term mentioned in a drug product label represents a true adverse drug event or an incidental mention.
This is the production model used by [OnSIDES](https://github.com/tatonetti-lab/onsides), an international database of adverse drug events extracted from product labels across four countries (USA, EU, UK, Japan).
## Model Details
- **Base model**: [microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract)
- **Task**: Binary text classification (is_event vs. not_event)
- **Architecture**: PubMedBERT + Dropout(0.5) + Linear(768, 2)
- **Training data**: 200 manually curated FDA drug labels from [Demner-Fushman et al.](https://pubmed.ncbi.nlm.nih.gov/29381145/), with MedDRA term matches labeled as adverse events or incidental mentions
- **Sections**: Trained jointly on Adverse Reactions (AR), Boxed Warnings (BW), and Warnings & Precautions (WP)
- **Training details**: Learning rate 1e-6, batch size 32, max sequence length 256, 125-word context window, early stopping with patience 4
## Performance
**Held-out test set** (80/10/10 drug-level split of 200 manually annotated FDA labels):
| Section | F1 | Precision | Recall | AUROC |
|---|---|---|---|---|
| Adverse Reactions | 0.942 | 0.962 | 0.922 | 0.996 |
| Boxed Warning | 0.901 | 0.977 | 0.835 | 0.996 |
| Warnings & Precautions | 0.880 | 0.851 | 0.911 | 0.995 |
**Independent hold-out** (30 manually annotated FDA labels, not used in training or threshold tuning):
| Section | F1 | Precision | Recall | AUROC |
|---|---|---|---|---|
| Adverse Reactions | 0.847 | 0.871 | 0.825 | 0.965 |
| Boxed Warning | 0.736 | 1.000 | 0.582 | 0.988 |
| Warnings & Precautions | 0.756 | 0.789 | 0.725 | 0.973 |
**TAC 2017 benchmark**: F1 = 89.87 (state of the art).
## Usage
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
tokenizer = AutoTokenizer.from_pretrained("tatonettilab/onsides-bert")
model = AutoModelForSequenceClassification.from_pretrained("tatonettilab/onsides-bert")
model.eval()
text = "Patients receiving EXAMPLE DRUG reported nausea, headache, and dizziness."
inputs = tokenizer(text, return_tensors="pt", max_length=256, truncation=True, padding="max_length")
with torch.no_grad():
outputs = model(**inputs)
# outputs.logits shape: (batch_size, 2)
# Column 0 = not_event score, Column 1 = is_event score
predicted_class = outputs.logits.argmax(dim=1).item()
print("is_event" if predicted_class == 1 else "not_event")
```
### Note on ReLU
The OnSIDES training pipeline applies a ReLU activation after the classification head.
The standard `BertForSequenceClassification` used here does not include that ReLU. For
simple classification (argmax), this makes no difference. If you are applying the
threshold-based scoring used in the OnSIDES pipeline, apply ReLU to the logits first:
```python
import torch.nn.functional as F
scores = F.relu(outputs.logits)
```
### Input Format
The model expects text constructed from drug label sections with MedDRA term context. In the OnSIDES pipeline, each input is a window of up to 125 words surrounding a candidate MedDRA term match, with the event term and source section prepended. See the [OnSIDES repository](https://github.com/tatonetti-lab/onsides) for the full text construction pipeline.
### Recommended Thresholds
For the OnSIDES v3.2.0 database, section-specific thresholds were applied to the ReLU-activated logit scores:
| Section | Threshold |
|---|---|
| Adverse Reactions | 0.6926 |
| Boxed Warning | 0.8713 |
| Warnings & Precautions | 0.5878 |
## Citation
```bibtex
@article{tanaka2025onsides,
title={OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models},
author={Tanaka, Yutaro and Chen, Hsin Yi and Belloni, Payal and Gisladottir, Undina and Kefeli, Jaden and Patterson, Joshua and Srinivasan, Ashwin and Zietz, Michael and Sirdeshmukh, Gaurav and Berkowitz, Jacob and LaRow Brown, Kathleen and Tatonetti, Nicholas P},
journal={Med},
year={2025},
publisher={Elsevier},
doi={10.1016/j.medj.2025.100642}
}
```
## License
MIT License. See the [OnSIDES repository](https://github.com/tatonetti-lab/onsides) for full details.
|