|
|
--- |
|
|
language: nl |
|
|
license: mit |
|
|
pipeline_tag: text-classification |
|
|
inference: false |
|
|
--- |
|
|
|
|
|
# A-PROOF ICF-domains Classification |
|
|
|
|
|
## Description |
|
|
|
|
|
A fine-tuned multi-label classification model that detects 9 [WHO-ICF](https://www.who.int/standards/classifications/international-classification-of-functioning-disability-and-health) domains in clinical text in Dutch. The model is based on a pre-trained Dutch medical language model ([link to be added]()), a RoBERTa model, trained from scratch on clinical notes of the Amsterdam UMC. |
|
|
|
|
|
## ICF domains |
|
|
The model can detect 9 domains, which were chosen due to their relevance to recovery from COVID-19: |
|
|
|
|
|
ICF code | Domain | name in repo |
|
|
---|---|--- |
|
|
b440 | Respiration functions | ADM |
|
|
b140 | Attention functions | ATT |
|
|
d840-d859 | Work and employment | BER |
|
|
b1300 | Energy level | ENR |
|
|
d550 | Eating | ETN |
|
|
d450 | Walking | FAC |
|
|
b455 | Exercise tolerance functions | INS |
|
|
b530 | Weight maintenance functions | MBW |
|
|
b152 | Emotional functions | STM |
|
|
|
|
|
## Intended uses and limitations |
|
|
- The model was fine-tuned (trained, validated and tested) on medical records from the Amsterdam UMC (the two academic medical centers of Amsterdam). It might perform differently on text from a different hospital or text from non-hospital sources (e.g. GP records). |
|
|
- The model was fine-tuned with the [Simple Transformers](https://simpletransformers.ai/) library. This library is based on Transformers but the model cannot be used directly with Transformers `pipeline` and classes; doing so would generate incorrect outputs. For this reason, the API on this page is disabled. |
|
|
|
|
|
## How to use |
|
|
To generate predictions with the model, use the [Simple Transformers](https://simpletransformers.ai/) library: |
|
|
``` |
|
|
from simpletransformers.classification import MultiLabelClassificationModel |
|
|
|
|
|
model = MultiLabelClassificationModel( |
|
|
'roberta', |
|
|
'CLTL/icf-domains', |
|
|
use_cuda=False, |
|
|
) |
|
|
|
|
|
example = 'Nu sinds 5-6 dagen progressieve benauwdheidsklachten (bij korte stukken lopen al kortademig), terwijl dit eerder niet zo was.' |
|
|
predictions, raw_outputs = model.predict([example]) |
|
|
``` |
|
|
The predictions look like this: |
|
|
``` |
|
|
[[1, 0, 0, 0, 0, 1, 1, 0, 0]] |
|
|
``` |
|
|
The indices of the multi-label stand for: |
|
|
``` |
|
|
[ADM, ATT, BER, ENR, ETN, FAC, INS, MBW, STM] |
|
|
``` |
|
|
In other words, the above prediction corresponds to assigning the labels ADM, FAC and INS to the example sentence. |
|
|
|
|
|
The raw outputs look like this: |
|
|
``` |
|
|
[[0.51907885 0.00268032 0.0030862 0.03066113 0.00616694 0.64720929 |
|
|
0.67348498 0.0118863 0.0046311 ]] |
|
|
``` |
|
|
For this model, the threshold at which the prediction for a label flips from 0 to 1 is **0.5**. |
|
|
|
|
|
## Training data |
|
|
- The training data consists of clinical notes from medical records (in Dutch) of the Amsterdam UMC. Due to privacy constraints, the data cannot be released. |
|
|
- The annotation guidelines used for the project can be found [here](https://github.com/cltl/a-proof-zonmw/tree/main/resources/annotation_guidelines). |
|
|
|
|
|
## Training procedure |
|
|
The default training parameters of Simple Transformers were used, including: |
|
|
- Optimizer: AdamW |
|
|
- Learning rate: 4e-5 |
|
|
- Num train epochs: 1 |
|
|
- Train batch size: 8 |
|
|
- Threshold: 0.5 |
|
|
|
|
|
## Evaluation results |
|
|
The evaluation is done on a sentence-level (the classification unit) and on a note-level (the aggregated unit which is meaningful for the healthcare professionals). |
|
|
|
|
|
### Sentence-level |
|
|
| | ADM | ATT | BER | ENR | ETN | FAC | INS | MBW | STM |
|
|
|---|---|---|---|---|---|---|---|---|--- |
|
|
precision | 0.98 | 0.98 | 0.56 | 0.96 | 0.92 | 0.84 | 0.89 | 0.79 | 0.70 |
|
|
recall | 0.49 | 0.41 | 0.29 | 0.57 | 0.49 | 0.71 | 0.26 | 0.62 | 0.75 |
|
|
F1-score | 0.66 | 0.58 | 0.35 | 0.72 | 0.63 | 0.76 | 0.41 | 0.70 | 0.72 |
|
|
support | 775 | 39 | 54 | 160 | 382 | 253 | 287 | 125 | 181 |
|
|
|
|
|
### Note-level |
|
|
| | ADM | ATT | BER | ENR | ETN | FAC | INS | MBW | STM |
|
|
|---|---|---|---|---|---|---|---|---|--- |
|
|
precision | 1.0 | 1.0 | 0.66 | 0.96 | 0.95 | 0.84 | 0.95 | 0.87 | 0.80 |
|
|
recall | 0.89 | 0.56 | 0.44 | 0.70 | 0.72 | 0.89 | 0.46 | 0.87 | 0.87 |
|
|
F1-score | 0.94 | 0.71 | 0.50 | 0.81 | 0.82 | 0.86 | 0.61 | 0.87 | 0.84 |
|
|
support | 231 | 27 | 34 | 92 | 165 | 95 | 116 | 64 | 94 |
|
|
|
|
|
## Authors and references |
|
|
### Authors |
|
|
Jenia Kim, Piek Vossen |
|
|
|
|
|
### References |
|
|
TBD |
