---
language:
- en
- hi
- kn
license: apache-2.0
tags:
- rag
- hallucination-detection
- xlm-roberta
- nli
- multilingual
- fact-checking
- text-classification
pipeline_tag: text-classification
widget:
- text: "Antigravity is an AI development platform."
  text_pair: "Antigravity is an AI development platform created by Deepmind engineers."
- text: "The deadline is 30 June 2026."
  text_pair: "Antigravity is an AI development platform created by Deepmind engineers."
---

# ClaimLens: Multilingual RAG Hallucination Verifier

**ClaimLens** is a fine-tuned multilingual (`English`, `Hindi`, `Kannada`) claim-level hallucination verifier for Retrieval-Augmented Generation (RAG) pipelines based on `xlm-roberta-base`.

It evaluates generated answers claim-by-claim against retrieved context passages and classifies each into one of three factual grounding states:
- **`Correct` (Label 0):** Retrieved evidence entails the claim.
- **`Incorrect` (Label 1):** Retrieved evidence contradicts the claim.
- **`Unverified` (Label 2):** Evidence neither confirms nor denies the claim (unsupported / baseless).

---

## Model Architecture & Training Protocol

* **Base Model:** `xlm-roberta-base`
* **Task:** Sentence-Pair Classification (`<s> claim </s></s> evidence </s>`)
* **Two-Stage Supervised Fine-Tuning:**
  1. **Stage A (NLI Warm-up):** Trained on 60,000 NLI pairs (20k each for English, Hindi, and Kannada from `facebook/xnli` and `Divyanshu/indicxnli`).
  2. **Stage B (RAG Fine-Tuning):** Fine-tuned on RAGTruth claim-level data (English) + machine-translated Hindi and Kannada subsets, using **Class-Weighted Cross-Entropy** to mitigate the ~90% majority class skew.

---

## Evaluation Results

Evaluated against the RAG test sets across languages:

| Metric | English (`en`) | Hindi (`hi`) | Kannada (`kn`) |
| :--- | :---: | :---: | :---: |
| **Macro-F1** | **0.4888** | **0.6086** | **0.5662** |
| **Hallucination F1** | **0.3676** | **0.8162** | **0.8057** |
| **Accuracy** | **76.61%** | **61.00%** | **56.67%** |
| **IndicGLUE WNLI (Hindi)** | — | **94.37%** | — |

*Significant performance improvement over Majority Class (0.1667) and TF-IDF baselines (0.4059 - 0.4270).*

---

## Quickstart & Usage

### Using Hugging Face Transformers

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "FaustDerLegende/RAGVERIFIER"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id)
model.eval()

claim = "Antigravity is an AI development platform."
evidence = "Antigravity is an AI development platform created by Deepmind engineers."

# Tokenize sentence-pair (claim, evidence)
inputs = tokenizer(
    claim,
    evidence,
    truncation="only_second",
    max_length=384,
    return_tensors="pt"
)

with torch.no_grad():
    logits = model(**inputs).logits
    probs = torch.softmax(logits, dim=-1)[0]

id2label = model.config.id2label
for i, prob in enumerate(probs):
    print(f"{id2label[i]}: {prob.item():.4f}")

predicted_class = id2label[torch.argmax(probs).item()]
print(f"
Final Verdict: {predicted_class}")
```

### Using Hugging Face Pipeline

```python
from transformers import pipeline

classifier = pipeline(
    "text-classification",
    model="FaustDerLegende/RAGVERIFIER",
    tokenizer="FaustDerLegende/RAGVERIFIER"
)

result = classifier({
    "text": "Antigravity is an AI development platform.",
    "text_pair": "Antigravity is an AI development platform created by Deepmind engineers."
})
print(result)
```

---

## Citation & Acknowledgment

Developed as part of the **ClaimLens** Multilingual Hallucination Detection project, adapting datasets from [RAGTruth (Niu et al.)](https://github.com/ParticleMedia/RAGTruth), [IndicXNLI (Aggarwal et al.)](https://huggingface.co/datasets/Divyanshu/indicxnli), and [IndicGLUE](https://huggingface.co/datasets/ai4bharat/indic_glue).