|
|
--- |
|
|
pipeline_tag: text-classification |
|
|
language: |
|
|
- nl |
|
|
base_model: |
|
|
- intfloat/multilingual-e5-small |
|
|
license: mit |
|
|
--- |
|
|
# Model Card |
|
|
|
|
|
This model is a fine-tuned version of [intfloat/multilingual-e5-small](https://huggingface.co/intfloat/multilingual-e5-small). It was fine-tuned on [Factrank](https://github.com/lejafar/FactRank/tree/master/factrank) data with additional machine annotated data from Dutch and Belgian parliamentary proceedings. |
|
|
|
|
|
The primary goal of this model is to determine whether a given statement warrants fact-checking. It does **not** determine whether the statement is factually correct. |
|
|
|
|
|
1 label is given: FR, FNR or NF. |
|
|
|
|
|
- **FR**: Factual Relevant (the statement is fact-checkable and requires verification) |
|
|
- **FNR**: Factual, Not Relevant (the statement can be fact-checked, but the wider relevance is lower) |
|
|
- **NF**: Not Factual (the statement does not contain information for fact-checking) |
|
|
|
|
|
**Examples**: |
|
|
- **FR**: *Toch blijkt uit cijfers van Flanders Investment & Trade dat ons handel met het Verenigd Koninkrijk opnieuw op het niveau ligt van voor de brexit.* |
|
|
- **FNR**: *Ayleen werd opgelicht via dating fraude door de Tinder Swindler: "Het zijn net vampiers."* |
|
|
- **NF**: *Het heeft weinig zin om zomaar een aantal maatregelen te tonen.* |
|
|
|
|
|
**Supported language**: Dutch |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import pipeline |
|
|
from transformers import AutoTokenizer, AutoModelForSequenceClassification, AutoConfig |
|
|
from huggingface_hub import login |
|
|
|
|
|
config = AutoConfig.from_pretrained("textgain/FactRank_e5_small") |
|
|
tokenizer = AutoTokenizer.from_pretrained("textgain/FactRank_e5_small") |
|
|
model = AutoModelForSequenceClassification.from_pretrained("textgain/FactRank_e5_small", config=config) |
|
|
pipe = pipeline(model=model, tokenizer=tokenizer, task="text-classification") |
|
|
|
|
|
|
|
|
sample_texts = [ |
|
|
"In een wereld die steeds digitaler wordt, moeten we het ook makkelijker maken om de controle over je financiën te hebben.", |
|
|
"Ik wil helemaal geen haren tussen u en de heer De Cock leggen.", |
|
|
"Je kunt van mening verschillen over welk gevolg je daaraan moet verbinden.", |
|
|
"We hebben 4.500 nieuwe kankergevallen in Nederland per jaar als gevolg van alcoholgebruik.", |
|
|
"Alcoholgebruik kost de samenleving 2 tot 4 miljard euro.", |
|
|
"Dus kan de minister daar vandaag wat meer over zeggen?" |
|
|
] |
|
|
|
|
|
results = pipe(sample_texts) |
|
|
predicted_labels = [res["label"] for res in results] |
|
|
``` |
|
|
|
|
|
## Interpretation of Results |
|
|
**Factors Influencing the Label:** |
|
|
- **Subjective Evaluation**: The presence of evaluations such as "interesting", "surprising", "incredible" might push the model towards predicting NF. |
|
|
- **Research**: The mention of research or studies pushes the model to consider the statement as a verifiable fact. |
|
|
- **Context**: Statements made in certain contexts may be more likely to get an FR label, e.g. statements about health and medicine. |
|
|
|
|
|
## Training Details |
|
|
The model was trained on a total of 13 786 data samples. |
|
|
|
|
|
Parameters: |
|
|
```python |
|
|
num_epochs = 5 |
|
|
batch_size = 32 |
|
|
learning_rate = 1e-5 |
|
|
dropout = 0.5 |
|
|
gradient_accumulation_steps = 4 |
|
|
``` |
|
|
|
|
|
## Acknowledgment |
|
|
|
|
|
<img src="https://benedmo.eu/wp-content/themes/benedmo/img/logo.svg" alt="BENEDMO Logo" width="200"> |
|
|
|
|
|
This transformer was made in the context of the [BENEDMO](https://www.benedmo.eu) project. BENEDMO brings together a network of expertise on disinformation and fact-checking. Through a Flemish-Dutch collaboration, BENEDMO aims to address the impact and challenges of disinformation. |
|
|
|
|
|
[BENEDMO](https://www.benedmo.eu) has received funding from the European Union under Grant Agreement number 101158277-BENEDMO-2023-DEPLOY-04. |
|
|
|
|
|
|
