| --- |
| language: |
| - en |
| - hi |
| - kn |
| license: apache-2.0 |
| tags: |
| - rag |
| - hallucination-detection |
| - xlm-roberta |
| - nli |
| - multilingual |
| - fact-checking |
| - text-classification |
| pipeline_tag: text-classification |
| widget: |
| - text: "Antigravity is an AI development platform." |
| text_pair: "Antigravity is an AI development platform created by Deepmind engineers." |
| - text: "The deadline is 30 June 2026." |
| text_pair: "Antigravity is an AI development platform created by Deepmind engineers." |
| --- |
| |
| # ClaimLens: Multilingual RAG Hallucination Verifier |
|
|
| **ClaimLens** is a fine-tuned multilingual (`English`, `Hindi`, `Kannada`) claim-level hallucination verifier for Retrieval-Augmented Generation (RAG) pipelines based on `xlm-roberta-base`. |
|
|
| It evaluates generated answers claim-by-claim against retrieved context passages and classifies each into one of three factual grounding states: |
| - **`Correct` (Label 0):** Retrieved evidence entails the claim. |
| - **`Incorrect` (Label 1):** Retrieved evidence contradicts the claim. |
| - **`Unverified` (Label 2):** Evidence neither confirms nor denies the claim (unsupported / baseless). |
|
|
| --- |
|
|
| ## Model Architecture & Training Protocol |
|
|
| * **Base Model:** `xlm-roberta-base` |
| * **Task:** Sentence-Pair Classification (`<s> claim </s></s> evidence </s>`) |
| * **Two-Stage Supervised Fine-Tuning:** |
| 1. **Stage A (NLI Warm-up):** Trained on 60,000 NLI pairs (20k each for English, Hindi, and Kannada from `facebook/xnli` and `Divyanshu/indicxnli`). |
| 2. **Stage B (RAG Fine-Tuning):** Fine-tuned on RAGTruth claim-level data (English) + machine-translated Hindi and Kannada subsets, using **Class-Weighted Cross-Entropy** to mitigate the ~90% majority class skew. |
|
|
| --- |
|
|
| ## Evaluation Results |
|
|
| Evaluated against the RAG test sets across languages: |
|
|
| | Metric | English (`en`) | Hindi (`hi`) | Kannada (`kn`) | |
| | :--- | :---: | :---: | :---: | |
| | **Macro-F1** | **0.4888** | **0.6086** | **0.5662** | |
| | **Hallucination F1** | **0.3676** | **0.8162** | **0.8057** | |
| | **Accuracy** | **76.61%** | **61.00%** | **56.67%** | |
| | **IndicGLUE WNLI (Hindi)** | — | **94.37%** | — | |
|
|
| *Significant performance improvement over Majority Class (0.1667) and TF-IDF baselines (0.4059 - 0.4270).* |
|
|
| --- |
|
|
| ## Quickstart & Usage |
|
|
| ### Using Hugging Face Transformers |
|
|
| ```python |
| import torch |
| from transformers import AutoTokenizer, AutoModelForSequenceClassification |
| |
| model_id = "FaustDerLegende/RAGVERIFIER" |
| tokenizer = AutoTokenizer.from_pretrained(model_id) |
| model = AutoModelForSequenceClassification.from_pretrained(model_id) |
| model.eval() |
| |
| claim = "Antigravity is an AI development platform." |
| evidence = "Antigravity is an AI development platform created by Deepmind engineers." |
| |
| # Tokenize sentence-pair (claim, evidence) |
| inputs = tokenizer( |
| claim, |
| evidence, |
| truncation="only_second", |
| max_length=384, |
| return_tensors="pt" |
| ) |
| |
| with torch.no_grad(): |
| logits = model(**inputs).logits |
| probs = torch.softmax(logits, dim=-1)[0] |
| |
| id2label = model.config.id2label |
| for i, prob in enumerate(probs): |
| print(f"{id2label[i]}: {prob.item():.4f}") |
| |
| predicted_class = id2label[torch.argmax(probs).item()] |
| print(f" |
| Final Verdict: {predicted_class}") |
| ``` |
|
|
| ### Using Hugging Face Pipeline |
|
|
| ```python |
| from transformers import pipeline |
| |
| classifier = pipeline( |
| "text-classification", |
| model="FaustDerLegende/RAGVERIFIER", |
| tokenizer="FaustDerLegende/RAGVERIFIER" |
| ) |
| |
| result = classifier({ |
| "text": "Antigravity is an AI development platform.", |
| "text_pair": "Antigravity is an AI development platform created by Deepmind engineers." |
| }) |
| print(result) |
| ``` |
|
|
| --- |
|
|
| ## Citation & Acknowledgment |
|
|
| Developed as part of the **ClaimLens** Multilingual Hallucination Detection project, adapting datasets from [RAGTruth (Niu et al.)](https://github.com/ParticleMedia/RAGTruth), [IndicXNLI (Aggarwal et al.)](https://huggingface.co/datasets/Divyanshu/indicxnli), and [IndicGLUE](https://huggingface.co/datasets/ai4bharat/indic_glue). |
|
|
