| --- |
| license: gpl-3.0 |
| datasets: |
| - DT4H/CardioCCC |
| language: |
| - ro |
| - it |
| - nl |
| - es |
| - en |
| - cs |
| - sv |
| base_model: |
| - FacebookAI/xlm-roberta-large |
| pipeline_tag: token-classification |
| tags: |
| - medical |
| --- |
| |
| Training and inference using [CardioNER](https://github.com/DataTools4Heart/CardioNER). |
|
|
| Settings |
| * Optimizer: AdamW |
| * Learning rate: 2e-5, with class-weights |
| * Decay rate: 1e-4 |
| * batch-size: 16 |
| * 10-fold stratified CV, with 10 epochs for the statistics below |
| * Architecture: EuroBERT + 3-layer dense head. |
| * Output: multilabel with probas for DISEASE, MEDICATION, PROCEDURE, SYMPTOM. |
| * Chunking: centered around the span of interest |
|
|
| Note: for inference we use ```poetry run python -m cardioner.main --inferency_only..``` [CardioNER](https://github.com/DataTools4Heart/CardioNER) with ```--pipe=dt4h```. |
|
|
| # Performance on internal hold-outs |
|
|
| Use with caution/not standalone: performance may be considerably less on external datasets. |
|
|
| ## Combined |
|
|
| | Setting | Averaging | Precision | Recall | F1 | |
| | ------- | --------- | ------------- | ------------- | ------------- | |
| | Strict | Micro | 0.731 ± 0.008 | 0.710 ± 0.007 | 0.721 ± 0.007 | |
| | Strict | Macro | 0.760 ± 0.007 | 0.735 ± 0.008 | 0.749 ± 0.007 | |
| | Relaxed | Micro | 0.875 ± 0.005 | 0.849 ± 0.008 | 0.864 ± 0.005 | |
| | Relaxed | Macro | 0.885 ± 0.007 | 0.860 ± 0.007 | 0.872 ± 0.007 | |
|
|
| ## Strict |
|
|
| | Category | Precision (mean ± std) | Recall (mean ± std) | F1 (mean ± std) | |
| | ---------- | ---------------------- | ------------------- | --------------- | |
| | DISEASE | 0.721 ± 0.010 | 0.705 ± 0.007 | 0.713 ± 0.009 | |
| | MEDICATION | 0.884 ± 0.011 | 0.847 ± 0.015 | 0.862 ± 0.012 | |
| | PROCEDURE | 0.754 ± 0.011 | 0.711 ± 0.010 | 0.730 ± 0.010 | |
| | SYMPTOM | 0.685 ± 0.008 | 0.680 ± 0.007 | 0.683 ± 0.009 | |
|
|
| ## Relaxed |
|
|
| | Category | Precision (mean ± std) | Recall (mean ± std) | F1 (mean ± std) | |
| | ---------- | ---------------------- | ------------------- | --------------- | |
| | DISEASE | 0.877 ± 0.009 | 0.856 ± 0.011 | 0.868 ± 0.008 | |
| | MEDICATION | 0.934 ± 0.011 | 0.895 ± 0.016 | 0.915 ± 0.012 | |
| | PROCEDURE | 0.885 ± 0.008 | 0.835 ± 0.008 | 0.858 ± 0.007 | |
| | SYMPTOM | 0.850 ± 0.009 | 0.845 ± 0.007 | 0.845 ± 0.007 | |
|
|
|
|
|
|
