|
|
--- |
|
|
language: sr |
|
|
metrics: |
|
|
- accuracy |
|
|
- precision |
|
|
- recall |
|
|
- f1 |
|
|
base_model: microsoft/deberta-v3-large |
|
|
--- |
|
|
# srbNLI: Serbian Natural Language Inference Model |
|
|
|
|
|
## Model Overview |
|
|
srbNLI is a fine-tuned Natural Language Inference (NLI) model for Serbian, created by adapting the SciFact dataset. The model is based on state-of-the-art transformer architectures. It is trained to recognize relationships between claims and evidence in Serbian text, with applications in scientific claim verification and potential expansion to broader claim verification tasks. |
|
|
|
|
|
## Key Details |
|
|
- **Model Type**: Transformer-based |
|
|
- **Language**: Serbian |
|
|
- **Task**: Natural Language Inference (NLI), Textual Entailment, Claim Verification |
|
|
- **Dataset**: srbSciFact (automatically translated SciFact dataset) |
|
|
- **Fine-tuning**: Fine-tuned on Serbian NLI data (support, contradiction, and neutral categories). |
|
|
- **Metrics**: Accuracy, Precision, Recall, F1-score |
|
|
|
|
|
## Motivation |
|
|
This model addresses the lack of NLI datasets and models for Serbian, a low-resource language. It provides a tool for textual entailment and claim verification, especially for scientific claims, with broader potential for misinformation detection and automated fact-checking. |
|
|
|
|
|
## Training |
|
|
- **Base Models Used**: DeBERTa-v3-large |
|
|
- **Training Data**: Automatically translated SciFact dataset |
|
|
- **Fine-tuning**: Conducted on a single DGX NVIDIA A100 GPU (40 GB) |
|
|
- **Hyperparameters**: Optimized learning rate, batch size, weight decay, epochs, and early stopping |
|
|
|
|
|
## Evaluation |
|
|
The model was evaluated using standard NLI metrics (accuracy, precision, recall, F1-score). It was also compared to the GPT-4o model for generalization capabilities. |
|
|
|
|
|
## Use Cases |
|
|
- **Claim Verification**: Scientific claims and general domain claims in Serbian |
|
|
- **Misinformation Detection**: Identifying contradictions or support between claims and evidence |
|
|
- **Cross-lingual Applications**: Potential for cross-lingual claim verification with multilingual models |
|
|
|
|
|
## Future Work |
|
|
- Improving accuracy with human-corrected translations and Serbian-specific datasets |
|
|
- Expanding to general-domain claim verification |
|
|
- Enhancing multilingual NLI capabilities |
|
|
|
|
|
## Results Comparison |
|
|
|
|
|
The table below presents a comparison of the fine-tuned models (DeBERTa-v3-large, RoBERTa-large, BERTić, GPT-4o, and others) on the srbSciFact dataset, focusing on key metrics: Accuracy (Acc), Precision (P), Recall (R), and F1-score (F1). The models were evaluated on their ability to classify relationships between claims and evidence in Serbian text. |
|
|
|
|
|
| Model | Accuracy | Precision (P) | Recall (R) | F1-score (F1) | |
|
|
|----------------------|----------|---------------|------------|---------------| |
|
|
| **DeBERTa-v3-large** | 0.70 | 0.86 | 0.82 | 0.84 | |
|
|
| **RoBERTa-large** | 0.57 | 0.63 | 0.76 | 0.69 | |
|
|
| **BERTić (Serbian)** | 0.56 | 0.56 | 0.37 | 0.44 | |
|
|
| **GPT-4o (English)** | 0.66 | 0.70 | 0.77 | 0.78 | |
|
|
| **mDeBERTa-base** | 0.63 | 0.92 | 0.75 | 0.83 | |
|
|
| **XLM-RoBERTa-large** | 0.64 | 0.89 | 0.77 | 0.83 | |
|
|
| **mBERT-cased** | 0.48 | 0.76 | 0.50 | 0.60 | |
|
|
| **mBERT-uncased** | 0.57 | 0.45 | 0.61 | 0.52 | |
|
|
|
|
|
### Observations |
|
|
- **DeBERTa-v3-large** performed the best overall, with an accuracy of 0.70 and an F1-score of 0.84. |
|
|
- **RoBERTa-large** and **BERTić** showed lower performance, especially in recall, suggesting challenges in handling complex linguistic inference in Serbian. |
|
|
- **GPT-4o** outperforms all fine-tuned models in F1-score when the prompt is in English, but the **DeBERTa-v3-large** model slightly outperforms GPT-4o when the prompt is in Serbian. |
|
|
- **mDeBERTa-base** and **XLM-RoBERTa-large** exhibited strong cross-lingual performance, with F1-scores of 0.83 and 0.83, respectively. |
|
|
|
|
|
This demonstrates the potential of adapting advanced transformer models to Serbian while highlighting areas for future improvement, such as refining translations and expanding domain-specific data. |
|
|
--- |
|
|
|
