---
license: mit
base_model: microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext
tags:
- token-classification
- ner
- clinical-trials
- biomedical
---

# PubMedBERT/BiomedBERT fine-tuned for CHIA clinical trial eligibility NER

Fine-tuned `microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract-fulltext` for
named entity recognition on clinical trial eligibility criteria (CHIA corpus),
as part of a course NLP project comparing fine-tuned biomedical transformers
vs. GPT-4 prompting.

- **Code / full writeup**: https://github.com/jatinpsingh/NER_Clinical_Trial_Eligibility/tree/main/pmb
- **Base model**: PubMedBERT-base (renamed BiomedBERT), Gu et al. 2021
- **Task**: token classification, 31 labels (`O` + 15 entity types x B/I)
- **Data**: CHIA eligibility criteria, team's shared fixed split (10,006 train / 1,240 val / 1,163 test sentences -- fold 0 of a 10-fold CV scheme, see below)

## Hyperparameters (matching Li et al. 2022, Table 3)

| | |
|---|---|
| Learning rate | 5e-5 |
| Batch size | 8 |
| Epochs | 10 |
| Max sequence length | 256 |
| Adam epsilon | 1e-8 |

## Results

**This checkpoint's own fold** (entity-level, test set):

| | Precision | Recall | F1 |
|---|---|---|---|
| Strict (exact span match) | 0.639 | 0.674 | 0.656 |
| Relaxed (type + overlap match) | 0.750 | 0.791 | 0.770 |

**Full 10-fold cross-validation** (mean +/- std across all 10 folds -- the
number directly comparable to Li et al. 2022's own 10-fold-averaged
reporting; the weights hosted in this repo are from one of these 10 folds,
not a checkpoint averaged across them):

| | Precision | Recall | F1 |
|---|---|---|---|
| Strict | 0.657 +/- 0.013 | 0.682 +/- 0.021 | 0.669 +/- 0.013 |
| Relaxed | 0.758 +/- 0.014 | 0.787 +/- 0.024 | 0.772 +/- 0.015 |

Both this checkpoint's own score and the 10-fold mean exceed Li et al. 2022's
published PubMedBERT numbers on Chia (0.622 strict / 0.744 relaxed).

## Usage

```python
from transformers import AutoModelForTokenClassification, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("ptanwar/pubmedbert-chia-ner")
model = AutoModelForTokenClassification.from_pretrained("ptanwar/pubmedbert-chia-ner")
```

## Known limitations

- The hosted weights are one fold's model, not a checkpoint averaged/ensembled
  across the 10-fold CV run -- treat the single-fold numbers above as this
  specific checkpoint's performance, and the 10-fold mean +/- std as the more
  rigorous estimate of the approach's true performance.
- Weakest on rare/ambiguous types (`Mood`, `Observation`, `Reference_point`) -- see the full writeup for error analysis.