ZaandaTeika/RAGHal-large-en-v1

raghal-modernbert-large-en-v1

RAGHal — token-level hallucination detector for Retrieval-Augmented Generation (RAG).
Trained from answerdotai/ModernBERT-large with a newly initialized token-classification head.

Overview

The model marks answer tokens that are not supported by the given context. Predictions are aggregated into character spans of hallucinated text.

ModernBERT supports long context (up to 8192 tokens), so full RAG contexts can usually be scored in one pass.

Model details

Name	raghal-modernbert-large-en-v1
Architecture	ModernBERT-large token classification (2 labels: clean / hallucinated)
Base model	answerdotai/ModernBERT-large
Initialization	Pretrained encoder + newly initialized classifier head
Max context	8192 tokens
Language	English
Tasks	Hallucination / attribution detection for RAG answers

Training data (auto-annotated, no human train labels)

This model is not trained on RAGTruth human span labels.
We keep the original RAGTruth train answers and prompts, and produce automatic token-level hallucination spans with our own annotation stack. Human labels are used only for evaluation (RAGTruth test).

Corpus (answers unchanged):

Item	Detail
Source	RAGTruth train split
Responses	Original answers from RAGTruth generators: GPT-4, GPT-3.5-turbo, Mistral-7B-Instruct, Llama-2-7B/13B/70B-chat (no re-generation)
Inputs	15 090 responses (2 515 sources × 6 generators)
Prompt	Original RAGTruth source_info prompt (QA / Summary / Data2txt)
Train labels	Auto-annotated spans only
Final train size	14 633 samples after postprocessing

Tasks in the final train set:

Task	Task definition	Samples
QA	Answer from retrieved passages; mark unsupported answer spans	4 925
Summary	Summarize a document; mark unsupported summary spans	4 758
Data2txt	Generate text from structured JSON; mark unsupported claims	4 950
Validation	RAGTruth human test set	2 700

Automatic annotation pipeline

Faithfulness labeling (unsupported / contradicts SOURCE), not open-world factuality. Spans are written as [HAL]…[/HAL] tags, then converted to character offsets.

RAGTruth train responses
        │
        ▼
 GPT-OSS-120B  Pass 1   (T=0.6)
   task-specific system + few-shot prompts
        │
        ▼
 GPT-OSS-120B  Critic   (T=0.3)
   removal-only: drop false-positive tags, never add new ones
        │
        ├─ Summary only ──► DeBERTa-large-MNLI filter (entailment thr=0.5)
        │
        ▼
 Postprocess (snap spans to word boundaries; Data2txt: merge adjacent spans)
        │
        ▼
 Token-classification JSON  (prompt, answer, char-span labels)

Stage	Tool / model	Role
Pass 1 annotator	openai/gpt-oss-120b (vLLM)	Propose hallucinated spans
Critic	same model, stricter prompt	Remove over-tagged spans
NLI filter (Summary)	microsoft/deberta-large-mnli	Drop spans entailed by the source document
Inference runtime	vLLM, multi-GPU	Batch annotation
Span postprocess	custom rules	Word-boundary snap; merge adjacent Data2txt spans

Task-specific annotation configs:

Task	Prompt pack	Critic	NLI
QA	system + 5 human-gold few-shots (refusal / synthesis / contradiction)	yes	no
Summary	system + few-shots	yes	yes (DeBERTa-MNLI @ 0.5)
Data2txt	system + aligned Data2txt rules (null fields, subjective descriptors)	full re-annot (pass1 → critic)	no

Final mix = QA (system) + Summary (system + critic + NLI) + Data2txt (aligned critic re-annotation).
No manual span editing on the train set.

Training hyperparameters

Parameter	Value
Optimizer	AdamW
Peak learning rate	1e-5
LR schedule	warmup ratio 0.05 + cosine
Batch size	8 (DataParallel, 2× GPU)
Gradient accumulation	1
Max epochs	10
Eval	2× / epoch
Early stopping	patience 3 validations, min 6 epochs
Class weights	disabled (uniform CE)
Stopped at	~8.0 epochs
Best val metric	example-level Hal F1

Usage

pip install transformers torch

from transformers import AutoTokenizer, AutoModelForTokenClassification

repo = "YOUR_ORG/raghal-modernbert-large-en-v1"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForTokenClassification.from_pretrained(repo)

Format inputs as prompt + answer (context and question in the prompt; answer is the sequence to label) and aggregate token predictions into character spans.

Performance

Values: precision / recall / F1 (%).
Evaluated on RAGTruth test (2700) and zero-shot PsiloQA English test (1098).

RAGTruth test — example-level

Task	P	R	F1
QA	71.43	62.50	66.67
Summary	61.40	51.47	56.00
Data2txt	89.12	87.74	88.42
Whole	80.93	75.61	78.18

RAGTruth test — span-level

Task	P	R	F1
QA	70.54	55.34	62.02
Summary	64.83	31.82	42.68
Data2txt	54.16	54.61	54.38
Whole	61.29	50.11	55.13

PsiloQA English test (zero-shot)

Metric	Value
AP	76.32%
IoU	51.02%

Comparison

Benchmark	raghal (this)	lettucedect-large	ModernBERT-large SFT on PsiloQA en
RAGTruth ex F1 (whole)	78.18	79.22	57.09
RAGTruth span F1 (whole)	55.13	58.93	23.58
PsiloQA AP	76.32	71.72	83.88
PsiloQA IoU	51.02	47.13	67.23

The PsiloQA column is the same ModernBERT-large architecture trained only on PsiloQA English train (in-domain on PsiloQA, poor transfer to RAGTruth).

Limitations

• English-only.
• Train labels are automatic (LLM teacher + critic + optional NLI), not human gold — residual annotation noise is possible.
• Tuned for RAGTruth-style QA / Summary / Data2txt prompts.
• Summary span recall is the weakest subtask.
• Not a multilingual detector.

Citation

@inproceedings{modernbert,
  title={Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference},
  author={Warner, Benjamin and others},
  booktitle={ACL},
  year={2025}
}

@inproceedings{nie2024ragtruth,
  title={RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models},
  author={Nie, Fuxiang and Yao, Yufeng and Zhu, Jingheng and others},
  booktitle={ACL},
  year={2024},
}