| --- |
| license: gpl-3.0 |
| datasets: |
| - DT4H/CardioCCC |
| language: |
| - ro |
| - it |
| - nl |
| - es |
| - en |
| - cs |
| - sv |
| base_model: |
| - EuroBERT/EuroBERT-610m |
| pipeline_tag: token-classification |
| tags: |
| - medical |
| --- |
| |
| Trained using [CardioNER](https://github.com/DataTools4Heart/CardioNER). |
| Settings |
| * Optimizer: AdamW |
| * Learning rate: 2e-5, with class-weights |
| * Decay rate: 1e-4 |
| * batch-size: 16 |
| * 10-fold stratified CV, with 10 epochs for the statistics below |
| * Architecture: EuroBERT + 3-layer dense head. |
| * Output: multilabel with probas for DISEASE, MEDICATION, PROCEDURE, SYMPTOM. |
| * Chunking: centered around the span of interest |
|
|
| Note: for inference we use ```poetry run python -m cardioner.main --inferency_only..``` [CardioNER](https://github.com/DataTools4Heart/CardioNER) with ```--pipe=dt4h```. |
|
|
| # Performance on internal hold-outs |
|
|
| Use with caution/not standalone: performance may be considerably less on external datasets. |
|
|
| ## Strict |
|
|
|
|
| | Category | Precision | Recall | F1 | |
| | ---------- | ------------: | ------------: | ------------: | |
| | **Micro** | 0.706 ± 0.009 | 0.583 ± 0.005 | 0.640 ± 0.004 | |
| | **Macro** | 0.745 ± 0.008 | 0.616 ± 0.007 | 0.673 ± 0.008 | |
| | DISEASE | 0.671 ± 0.009 | 0.562 ± 0.007 | 0.611 ± 0.007 | |
| | MEDICATION | 0.909 ± 0.009 | 0.749 ± 0.016 | 0.821 ± 0.009 | |
| | PROCEDURE | 0.734 ± 0.010 | 0.587 ± 0.010 | 0.655 ± 0.007 | |
| | SYMPTOM | 0.666 ± 0.009 | 0.566 ± 0.009 | 0.611 ± 0.007 | |
|
|
| ## Relaxed |
|
|
| | Category | Precision | Recall | F1 | |
| | ---------- | ------------: | ------------: | ------------: | |
| | **Micro** | 0.876 ± 0.009 | 0.728 ± 0.007 | 0.796 ± 0.005 | |
| | **Macro** | 0.894 ± 0.007 | 0.738 ± 0.009 | 0.809 ± 0.005 | |
| | DISEASE | 0.872 ± 0.010 | 0.733 ± 0.012 | 0.794 ± 0.007 | |
| | MEDICATION | 0.955 ± 0.008 | 0.787 ± 0.016 | 0.861 ± 0.011 | |
| | PROCEDURE | 0.898 ± 0.009 | 0.717 ± 0.010 | 0.797 ± 0.006 | |
| | SYMPTOM | 0.846 ± 0.009 | 0.720 ± 0.011 | 0.779 ± 0.007 | |
|
|
|
|
