README.md

---
language:
- vi
library_name: transformers
pipeline_tag: text-classification
base_model: "uitnlp/CafeBERT"
datasets:
- tranthaihoa/vifactcheck
tags:
- vietnamese
- text-classification
- fact-checking
- claim-verification
- natural-language-inference
- vifactcheck
- gold-evidence
- eacl-2027
metrics:
- f1
- accuracy
---

# cafebert-ViFactCheck-GE

This model is `uitnlp/CafeBERT` fine-tuned for **VFC-GE** on ViFactCheck using the claim paired with **gold evidence**.

## Evaluation protocol

- Dataset size: 7,232 examples.
- Shared fixed stratified splits for FC and GE: 5,785 train / 723 development / 724 test.
- Labels: Supported, Refuted, and Not Enough Information.
- Fine-tuning seeds: [42, 22, 202].
- Training: 3 epoch(s), AdamW, learning rate 2e-05, weight decay 0.01, warmup ratio 0.1.
- Effective train batch size: **8** (hard-validated against every published run).
- Maximum sequence length: 256.
- Input mode: raw Vietnamese claim and passage.
- The claim is always preserved; only the second sequence (gold evidence) is truncated when the pair exceeds the encoder limit.
- Topic, author, outlet, URL and other source metadata are excluded from model inputs.
- No class weighting, resampling, retrieval model, sentence ranking, test-time model selection or external evidence is used.
- Checkpoints are selected by development Macro-F1. The representative published checkpoint is seed **202**, selected only by development Macro-F1.

## Results

Test metrics are reported as mean ± sample standard deviation over seeds [42, 22, 202].

| Metric | Mean ± std |
|---|---:|
| Test Macro-F1 | 0.8795 ± 0.0139 |
| Test accuracy | 0.8798 ± 0.0132 |
| Test macro precision | 0.8841 ± 0.0126 |
| Test macro recall | 0.8790 ± 0.0131 |
| Development Macro-F1 | 0.8810 ± 0.0030 |

### Per-seed results

|       seed |   dev_macro_f1 |   test_macro_f1 |   test_accuracy |   micro_batch_size |   gradient_accumulation_steps |
|-----------:|---------------:|----------------:|----------------:|-------------------:|------------------------------:|
|  22.000000 |       0.877927 |        0.864836 |        0.866022 |           8.000000 |                      1.000000 |
|  42.000000 |       0.881033 |        0.881194 |        0.881215 |           8.000000 |                      1.000000 |
| 202.000000 |       0.883915 |        0.892466 |        0.892265 |           8.000000 |                      1.000000 |

## Label mapping

```json
{
  "0": "supported",
  "1": "refuted",
  "2": "not_enough_information"
}
```

## Usage

```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_id = "BaoNhan/cafebert-ViFactCheck-GE"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSequenceClassification.from_pretrained(model_id)

claim = "Thông tin này đã được cơ quan chức năng xác nhận."
evidence = "Bài báo cung cấp bằng chứng liên quan đến phát biểu trên."
inputs = tokenizer(
    claim,
    evidence,
    return_tensors="pt",
    truncation="only_second",
    max_length=256,
)
with torch.no_grad():
    probabilities = model(**inputs).logits.softmax(dim=-1)[0]
predicted_id = int(probabilities.argmax())
print(model.config.id2label[predicted_id], probabilities.tolist())
```

## Files

- `aggregate_metrics.json`: aggregate metrics and training manifest.
- `artifacts/per_seed_results.csv`: one row per fine-tuning seed.
- `artifacts/seed_*_confusion_matrix.csv`: confusion matrix for each seed.
- `artifacts/seed_*_classification_report.json`: per-class metrics.
- `artifacts/seed_*_test_predictions.csv`: IDs, gold/predicted labels and probabilities; raw claims and passages are excluded.

## Limitations

ViFactCheck supplies the correct source article and therefore does not evaluate open-web evidence retrieval. **VFC-FC** can truncate relevant information in long articles and jointly measures verification plus robustness to irrelevant context. **VFC-GE** uses oracle gold evidence and must not be presented as a realistic end-to-end deployment setting. This model is a research classifier, not an automated arbiter of truth, and may produce confidently incorrect predictions.

## Dataset citation

```bibtex
@inproceedings{hoa2025vifactcheck,
  title={ViFactCheck: A New Benchmark Dataset and Methods for Multi-domain News Fact-Checking in Vietnamese},
  author={Hoa, Tran Thai and Duy, Tran Quang and Tran, Khanh Quoc and Nguyen, Kiet Van},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={39},
  number={1},
  pages={308--316},
  year={2025},
  doi={10.1609/aaai.v39i1.32008}
}
```