File size: 3,936 Bytes
095df10 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 | ---
language:
- en
- hi
- kn
license: apache-2.0
tags:
- rag
- hallucination-detection
- xlm-roberta
- nli
- multilingual
- fact-checking
- text-classification
pipeline_tag: text-classification
widget:
- text: "Antigravity is an AI development platform."
text_pair: "Antigravity is an AI development platform created by Deepmind engineers."
- text: "The deadline is 30 June 2026."
text_pair: "Antigravity is an AI development platform created by Deepmind engineers."
---
# ClaimLens: Multilingual RAG Hallucination Verifier
**ClaimLens** is a fine-tuned multilingual (`English`, `Hindi`, `Kannada`) claim-level hallucination verifier for Retrieval-Augmented Generation (RAG) pipelines based on `xlm-roberta-base`.
It evaluates generated answers claim-by-claim against retrieved context passages and classifies each into one of three factual grounding states:
- **`Correct` (Label 0):** Retrieved evidence entails the claim.
- **`Incorrect` (Label 1):** Retrieved evidence contradicts the claim.
- **`Unverified` (Label 2):** Evidence neither confirms nor denies the claim (unsupported / baseless).
---
## Model Architecture & Training Protocol
* **Base Model:** `xlm-roberta-base`
* **Task:** Sentence-Pair Classification (`<s> claim </s></s> evidence </s>`)
* **Two-Stage Supervised Fine-Tuning:**
1. **Stage A (NLI Warm-up):** Trained on 60,000 NLI pairs (20k each for English, Hindi, and Kannada from `facebook/xnli` and `Divyanshu/indicxnli`).
2. **Stage B (RAG Fine-Tuning):** Fine-tuned on RAGTruth claim-level data (English) + machine-translated Hindi and Kannada subsets, using **Class-Weighted Cross-Entropy** to mitigate the ~90% majority class skew.
---
## Evaluation Results
Evaluated against the RAG test sets across languages:
| Metric | English (`en`) | Hindi (`hi`) | Kannada (`kn`) |
| :--- | :---: | :---: | :---: |
| **Macro-F1** | **0.4888** | **0.6086** | **0.5662** |
| **Hallucination F1** | **0.3676** | **0.8162** | **0.8057** |
| **Accuracy** | **76.61%** | **61.00%** | **56.67%** |
| **IndicGLUE WNLI (Hindi)** | — | **94.37%** | — |
*Significant performance improvement over Majority Class (0.1667) and TF-IDF baselines (0.4059 - 0.4270).*
---
## Quickstart & Usage
### Using Hugging Face Transformers
```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model_id = "FaustDerLegende/RAGVERIFIER"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()
claim = "Antigravity is an AI development platform."
evidence = "Antigravity is an AI development platform created by Deepmind engineers."
# Tokenize sentence-pair (claim, evidence)
inputs = tokenizer(
claim,
evidence,
truncation="only_second",
max_length=384,
return_tensors="pt"
)
with torch.no_grad():
logits = model(**inputs).logits
probs = torch.softmax(logits, dim=-1)[0]
id2label = model.config.id2label
for i, prob in enumerate(probs):
print(f"{id2label[i]}: {prob.item():.4f}")
predicted_class = id2label[torch.argmax(probs).item()]
print(f"
Final Verdict: {predicted_class}")
```
### Using Hugging Face Pipeline
```python
from transformers import pipeline
classifier = pipeline(
"text-classification",
model="FaustDerLegende/RAGVERIFIER",
tokenizer="FaustDerLegende/RAGVERIFIER"
)
result = classifier({
"text": "Antigravity is an AI development platform.",
"text_pair": "Antigravity is an AI development platform created by Deepmind engineers."
})
print(result)
```
---
## Citation & Acknowledgment
Developed as part of the **ClaimLens** Multilingual Hallucination Detection project, adapting datasets from [RAGTruth (Niu et al.)](https://github.com/ParticleMedia/RAGTruth), [IndicXNLI (Aggarwal et al.)](https://huggingface.co/datasets/Divyanshu/indicxnli), and [IndicGLUE](https://huggingface.co/datasets/ai4bharat/indic_glue).
|