Create README.md

README.md

@@ -0,0 +1,67 @@
+---
+language: sr
+metrics:
+  - accuracy
+  - precision
+  - recall
+  - f1
+base_model: microsoft/deberta-v3-large
+---
+# srbNLI: Serbian Natural Language Inference Model
+
+## Model Overview
+srbNLI is a fine-tuned Natural Language Inference (NLI) model for Serbian, created by adapting the SciFact dataset. The model is based on state-of-the-art transformer architectures. It is trained to recognize relationships between claims and evidence in Serbian text, with applications in scientific claim verification and potential expansion to broader claim verification tasks.
+
+## Key Details
+- **Model Type**: Transformer-based
+- **Language**: Serbian
+- **Task**: Natural Language Inference (NLI), Textual Entailment, Claim Verification
+- **Dataset**: srbSciFact (automatically translated SciFact dataset)
+- **Fine-tuning**: Fine-tuned on Serbian NLI data (support, contradiction, and neutral categories).
+- **Metrics**: Accuracy, Precision, Recall, F1-score
+
+## Motivation
+This model addresses the lack of NLI datasets and models for Serbian, a low-resource language. It provides a tool for textual entailment and claim verification, especially for scientific claims, with broader potential for misinformation detection and automated fact-checking.
+
+## Training
+- **Base Models Used**: DeBERTa-v3-large
+- **Training Data**: Automatically translated SciFact dataset
+- **Fine-tuning**: Conducted on a single DGX NVIDIA A100 GPU (40 GB)
+- **Hyperparameters**: Optimized learning rate, batch size, weight decay, epochs, and early stopping
+
+## Evaluation
+The model was evaluated using standard NLI metrics (accuracy, precision, recall, F1-score). It was also compared to the GPT-4o model for generalization capabilities.
+
+## Use Cases
+- **Claim Verification**: Scientific claims and general domain claims in Serbian
+- **Misinformation Detection**: Identifying contradictions or support between claims and evidence
+- **Cross-lingual Applications**: Potential for cross-lingual claim verification with multilingual models
+
+## Future Work
+- Improving accuracy with human-corrected translations and Serbian-specific datasets
+- Expanding to general-domain claim verification
+- Enhancing multilingual NLI capabilities
+
+## Results Comparison
+
+The table below presents a comparison of the fine-tuned models (DeBERTa-v3-large, RoBERTa-large, BERTić, GPT-4o, and others) on the srbSciFact dataset, focusing on key metrics: Accuracy (Acc), Precision (P), Recall (R), and F1-score (F1). The models were evaluated on their ability to classify relationships between claims and evidence in Serbian text.
+
+| Model                | Accuracy | Precision (P) | Recall (R) | F1-score (F1) |
+|----------------------|----------|---------------|------------|---------------|
+| **DeBERTa-v3-large**  | 0.70     | 0.86          | 0.82       | 0.84          |
+| **RoBERTa-large**     | 0.57     | 0.63          | 0.76       | 0.69          |
+| **BERTić (Serbian)**  | 0.56     | 0.56          | 0.37       | 0.44          |
+| **GPT-4o (English)**  | 0.66     | 0.70          | 0.77       | 0.78          |
+| **mDeBERTa-base**     | 0.63     | 0.92          | 0.75       | 0.83          |
+| **XLM-RoBERTa-large** | 0.64     | 0.89          | 0.77       | 0.83          |
+| **mBERT-cased**       | 0.48     | 0.76          | 0.50       | 0.60          |
+| **mBERT-uncased**     | 0.57     | 0.45          | 0.61       | 0.52          |
+
+### Observations
+- **DeBERTa-v3-large** performed the best overall, with an accuracy of 0.70 and an F1-score of 0.84.
+- **RoBERTa-large** and **BERTić** showed lower performance, especially in recall, suggesting challenges in handling complex linguistic inference in Serbian.
+- **GPT-4o** outperforms all fine-tuned models in F1-score when the prompt is in English, but the **DeBERTa-v3-large** model slightly outperforms GPT-4o when the prompt is in Serbian.
+- **mDeBERTa-base** and **XLM-RoBERTa-large** exhibited strong cross-lingual performance, with F1-scores of 0.83 and 0.83, respectively.
+
+This demonstrates the potential of adapting advanced transformer models to Serbian while highlighting areas for future improvement, such as refining translations and expanding domain-specific data.
+---