README.md

---
license: gpl-3.0
language:
- en
metrics:
- accuracy
base_model: facebook/bart-large
---

# Model Card for ANGEL_pretrained
This model card provides detailed information about the ANGEL_pretrained model, designed for biomedical entity linking.

# Model Details

#### Model Description
- **Developed by:** Chanhwi Kim, Hyunjae Kim, Sihyeon Park, Jiwoo Lee, Mujeen Sung, Jaewoo Kang
- **Model type:** Generative Biomedical Entity Linking Model
- **Language(s):** English
- **License:** GPL-3.0
- **Finetuned from model:** BART-large (Base architecture)

#### Model Sources

- **Repository:** https://github.com/dmis-lab/ANGEL
- **Paper:** https://arxiv.org/pdf/2408.16493


# Direct Use
ANGEL_pretrained is pretrained with UMLS dataset.
We recommand to finetune this model to downstream dataset rather directly use.
If you still want to run the model on a single sample, no preprocessing is required. 
Simply execute the run_sample.sh script:

```bash
bash script/inference/run_sample.sh pretrained
```

To modify the sample with your own example, refer to the [Direct Use](https://github.com/dmis-lab/ANGEL?tab=readme-ov-file#direct-use) section in our GitHub repository.
If you're interested in training or evaluating the model, check out the [Fine-tuning](https://github.com/dmis-lab/ANGEL?tab=readme-ov-file#fine-tuning) section and [Evaluation](https://github.com/dmis-lab/ANGEL?tab=readme-ov-file#evaluation) section.


# Training Details

#### Training Data
The model was pretrained on the UMLS-2020-AA dataset.

#### Training Procedure
Positive-only Pre-training: Initial training using only positive examples, following the standard approach.

Negative-aware Training: Subsequent training incorporated negative examples to improve the model's discriminative capabilities.

# Evaluation

#### Testing Data
The model was evaluated using multiple biomedical datasets, including NCBI-disease, BC5CDR, COMETA, AAP, and MedMentions.
The fine-tuned scores have also been included.

#### Metrics
**Accuracy at Top-1 (Acc@1)**: Measures the percentage of times the model's top prediction matches the correct entity.

### Results

<table border="1" cellspacing="0" cellpadding="5" style="width: 100%; text-align: center; border-collapse: collapse; margin-left: 0;">
  <thead>
    <tr>
      <th style="text-align: center;"><b>Model</b></th>
      <th style="text-align: center;"><b>NCBI-disease</b></th>
      <th style="text-align: center;"><b>BC5CDR</b></th>
      <th style="text-align: center;"><b>COMETA</b></th>
      <th style="text-align: center;"><b>AAP</b></th>
      <th style="text-align: center;"><b>MedMentions<br>ST21pv</b></th>
      <th style="text-align: center;"><b>Average</b></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><b>GenBioEL_pretrained</b></td>
      <td>58.2</td>
      <td>33.1</td>
      <td>42.4</td>
      <td>50.6</td>
      <td>10.6</td>
      <td><b>39.0</b></td>
    </tr>
    <tr>
      <td><b>ANGEL_pretrained (Ours)</b></td>
      <td>64.6</td>
      <td>49.7</td>
      <td>46.8</td>
      <td>61.5</td>
      <td>18.2</td>
      <td><b>48.2</b></td>
    </tr>
    <tr>
      <td><b>GenBioEL_pt_ft</b></td>
      <td>91.0</td>
      <td>93.1</td>
      <td>80.9</td>
      <td>89.3</td>
      <td>70.7</td>
      <td><b>85.0</b></td>
    </tr>
    <tr>
      <td><b>ANGEL_pt_ft (Ours)</b></td>
      <td>92.8</td>
      <td>94.5</td>
      <td>82.8</td>
      <td>90.2</td>
      <td>73.3</td>
      <td><b>86.7</b></td>
    </tr>
  </tbody>
</table>

- In this table, "pt" refers to pre-training, where the model is trained on a large dataset (UMLS in this case), and "ft" refers to fine-tuning, where the model is further refined on specific datasets.

In the pre-training phase, **ANGEL** was trained using UMLS dataset entities that were similar to a given word based on TF-IDF scores but had different CUIs (Concept Unique Identifiers). 
This negative-aware pre-training approach improved its performance across the benchmarks, leading to an average score of 48.2, which is **9.2** points higher than the GenBioEL pre-trained model, which scored 39.0 on average.

The performance improvement continued during the fine-tuning phase. After fine-tuning, ANGEL achieved an average score of 86.7, surpassing the GenBioEL model's average score of 85.0, representing a further **1.7** point improvement. The ANGEL model consistently outperformed GenBioEL across all datasets in this phase.
The results demonstrate that the negative-aware training introduced by ANGEL not only enhances performance during pre-training but also carries over into fine-tuning, helping the model generalize better to unseen data.

# Citation
If you use the ANGEL_ncbi model, please cite:

```bibtex
@article{kim2024learning,
  title={Learning from Negative Samples in Generative Biomedical Entity Linking},
  author={Kim, Chanhwi and Kim, Hyunjae and Park, Sihyeon and Lee, Jiwoo and Sung, Mujeen and Kang, Jaewoo},
  journal={arXiv preprint arXiv:2408.16493},
  year={2024}
}
```

# Contact
For questions or issues, please contact chanhwi_kim@korea.ac.kr.