README.md

---
license: mit
language: en
tags:
  - adverse-drug-events
  - drug-safety
  - pharmacovigilance
  - text-classification
  - biomedical
  - PubMedBERT
datasets:
  - custom
base_model: microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract
pipeline_tag: text-classification
library_name: transformers
---

# OnSIDES-BERT: Adverse Drug Event Classifier

A fine-tuned [PubMedBERT](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract) model for classifying whether a medical term mentioned in a drug product label represents a true adverse drug event or an incidental mention.

This is the production model used by [OnSIDES](https://github.com/tatonetti-lab/onsides), an international database of adverse drug events extracted from product labels across four countries (USA, EU, UK, Japan).

## Model Details

- **Base model**: [microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract](https://huggingface.co/microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract)
- **Task**: Binary text classification (is_event vs. not_event)
- **Architecture**: PubMedBERT + Dropout(0.5) + Linear(768, 2)
- **Training data**: 200 manually curated FDA drug labels from [Demner-Fushman et al.](https://pubmed.ncbi.nlm.nih.gov/29381145/), with MedDRA term matches labeled as adverse events or incidental mentions
- **Sections**: Trained jointly on Adverse Reactions (AR), Boxed Warnings (BW), and Warnings & Precautions (WP)
- **Training details**: Learning rate 1e-6, batch size 32, max sequence length 256, 125-word context window, early stopping with patience 4

## Performance

**Held-out test set** (80/10/10 drug-level split of 200 manually annotated FDA labels):

| Section | F1 | Precision | Recall | AUROC |
|---|---|---|---|---|
| Adverse Reactions | 0.942 | 0.962 | 0.922 | 0.996 |
| Boxed Warning | 0.901 | 0.977 | 0.835 | 0.996 |
| Warnings & Precautions | 0.880 | 0.851 | 0.911 | 0.995 |

**Independent hold-out** (30 manually annotated FDA labels, not used in training or threshold tuning):

| Section | F1 | Precision | Recall | AUROC |
|---|---|---|---|---|
| Adverse Reactions | 0.847 | 0.871 | 0.825 | 0.965 |
| Boxed Warning | 0.736 | 1.000 | 0.582 | 0.988 |
| Warnings & Precautions | 0.756 | 0.789 | 0.725 | 0.973 |

**TAC 2017 benchmark**: F1 = 89.87 (state of the art).

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("tatonettilab/onsides-bert")
model = AutoModelForSequenceClassification.from_pretrained("tatonettilab/onsides-bert")
model.eval()

text = "Patients receiving EXAMPLE DRUG reported nausea, headache, and dizziness."
inputs = tokenizer(text, return_tensors="pt", max_length=256, truncation=True, padding="max_length")

with torch.no_grad():
    outputs = model(**inputs)

# outputs.logits shape: (batch_size, 2)
# Column 0 = not_event score, Column 1 = is_event score
predicted_class = outputs.logits.argmax(dim=1).item()
print("is_event" if predicted_class == 1 else "not_event")
```

### Note on ReLU

The OnSIDES training pipeline applies a ReLU activation after the classification head.
The standard `BertForSequenceClassification` used here does not include that ReLU. For
simple classification (argmax), this makes no difference. If you are applying the
threshold-based scoring used in the OnSIDES pipeline, apply ReLU to the logits first:

```python
import torch.nn.functional as F
scores = F.relu(outputs.logits)
```

### Input Format

The model expects text constructed from drug label sections with MedDRA term context. In the OnSIDES pipeline, each input is a window of up to 125 words surrounding a candidate MedDRA term match, with the event term and source section prepended. See the [OnSIDES repository](https://github.com/tatonetti-lab/onsides) for the full text construction pipeline.

### Recommended Thresholds

For the OnSIDES v3.2.0 database, section-specific thresholds were applied to the ReLU-activated logit scores:

| Section | Threshold |
|---|---|
| Adverse Reactions | 0.6926 |
| Boxed Warning | 0.8713 |
| Warnings & Precautions | 0.5878 |

## Citation

```bibtex
@article{tanaka2025onsides,
  title={OnSIDES database: Extracting adverse drug events from drug labels using natural language processing models},
  author={Tanaka, Yutaro and Chen, Hsin Yi and Belloni, Payal and Gisladottir, Undina and Kefeli, Jaden and Patterson, Joshua and Srinivasan, Ashwin and Zietz, Michael and Sirdeshmukh, Gaurav and Berkowitz, Jacob and LaRow Brown, Kathleen and Tatonetti, Nicholas P},
  journal={Med},
  year={2025},
  publisher={Elsevier},
  doi={10.1016/j.medj.2025.100642}
}
```

## License

MIT License. See the [OnSIDES repository](https://github.com/tatonetti-lab/onsides) for full details.