---
license: gpl-3.0
datasets:
- DT4H/CardioCCC
language:
- ro
- it
- nl
- es
- en
- cs
- sv
base_model:
- FacebookAI/xlm-roberta-large
pipeline_tag: token-classification
tags:
- medical
---

Training and inference using [CardioNER](https://github.com/DataTools4Heart/CardioNER). 

Settings
* Optimizer: AdamW
* Learning rate: 2e-5, with class-weights
* Decay rate: 1e-4
* batch-size: 16
* 10-fold stratified CV, with 10 epochs for the statistics below
* Architecture: EuroBERT + 3-layer dense head.
* Output: multilabel with probas for DISEASE, MEDICATION, PROCEDURE, SYMPTOM.
* Chunking: centered around the span of interest

Note: for inference we use ```poetry run python -m cardioner.main --inferency_only..``` [CardioNER](https://github.com/DataTools4Heart/CardioNER) with ```--pipe=dt4h```.

# Performance on internal hold-outs 

Use with caution/not standalone: performance may be considerably less on external datasets.

## Combined

| Setting | Averaging | Precision     | Recall        | F1            |
| ------- | --------- | ------------- | ------------- | ------------- |
| Strict  | Micro     | 0.731 ± 0.008 | 0.710 ± 0.007 | 0.721 ± 0.007 |
| Strict  | Macro     | 0.760 ± 0.007 | 0.735 ± 0.008 | 0.749 ± 0.007 |
| Relaxed | Micro     | 0.875 ± 0.005 | 0.849 ± 0.008 | 0.864 ± 0.005 |
| Relaxed | Macro     | 0.885 ± 0.007 | 0.860 ± 0.007 | 0.872 ± 0.007 |

## Strict

| Category   | Precision (mean ± std) | Recall (mean ± std) | F1 (mean ± std) |
| ---------- | ---------------------- | ------------------- | --------------- |
| DISEASE    | 0.721 ± 0.010          | 0.705 ± 0.007       | 0.713 ± 0.009   |
| MEDICATION | 0.884 ± 0.011          | 0.847 ± 0.015       | 0.862 ± 0.012   |
| PROCEDURE  | 0.754 ± 0.011          | 0.711 ± 0.010       | 0.730 ± 0.010   |
| SYMPTOM    | 0.685 ± 0.008          | 0.680 ± 0.007       | 0.683 ± 0.009   |

## Relaxed

| Category   | Precision (mean ± std) | Recall (mean ± std) | F1 (mean ± std) |
| ---------- | ---------------------- | ------------------- | --------------- |
| DISEASE    | 0.877 ± 0.009          | 0.856 ± 0.011       | 0.868 ± 0.008   |
| MEDICATION | 0.934 ± 0.011          | 0.895 ± 0.016       | 0.915 ± 0.012   |
| PROCEDURE  | 0.885 ± 0.008          | 0.835 ± 0.008       | 0.858 ± 0.007   |
| SYMPTOM    | 0.850 ± 0.009          | 0.845 ± 0.007       | 0.845 ± 0.007   |