landtml/d1-2B-RAG-Verifier

d1-2B RAG Verifier

A 2B-parameter verifier for retrieval-augmented generation. Give it the retrieved source, the user's request and the generated answer, and it returns the probability that every claim in the answer is supported by the source. Unsupported claims are what RAGTruth calls hallucinations, so this is response-level hallucination detection.

The verdict comes from a single forward pass with no text generation, which makes it cheap enough to run on every answer a RAG system produces. It is LiquidAI/d1-2B fine-tuned with LoRA on the RAGTruth training split.

Try it in the browser: paste a source and an answer, or pick one of the enterprise examples.

81.5 F1	0.936 AUROC	69%	~40 / s
RAGTruth test, response level	ranking of supported vs unsupported answers	of answers decided alone at 95% accuracy	answers per second on one RTX 5090; 144 ms each on a 12 GB consumer GPU

• Highest F1 below 8B parameters among the published fine-tuned detectors compared here: ahead of LettuceDetect-large (79.2) and the fine-tuned Llama-2-13B from the RAGTruth paper (78.7), a model 7 times larger.
• Within 2.4 F1 of RAG-HAT (Llama-3-8B, 83.9) at a quarter of its size.
• Catches 83.2% of hallucinated answers, the highest recall of the detectors above 75 F1.
• Calibrated scores. The probabilities track observed rates (expected calibration error 0.030), so thresholds and review-queue sizes behave as expected.

Results

RAGTruth test split: 2,700 responses, 900 each for question answering, data-to-text and summarization. A response counts as hallucinated if annotators marked any unsupported span in it.

Model	Params	QA F1	Data-to-text F1	Summary F1	Precision	Recall	F1
RAG-HAT (Llama-3-8B)	8B	74.8	91.6	67.6	87.3	80.8	83.9
d1-2B RAG Verifier (this model)	1.9B	72.0	89.9	64.3	79.8	83.2	81.5
LettuceDetect-large	396M	70.2	88.5	59.7	80.4	78.1	79.2
Llama-2-13B, fine-tuned (RAGTruth paper)	13B	68.2	88.1	59.1	76.9	80.7	78.7
LettuceDetect-base	150M	65.5	87.9	50.5	76.6	75.5	76.1
d1-2B, untrained	1.9B	58.4	84.6	27.2	83.6	61.6	70.9

The two d1 rows were measured for this release. The other rows are the numbers their authors report: Llama-2-13B from the RAGTruth paper (Table 5), RAG-HAT and LettuceDetect from the LettuceDetect paper (Table 2). Bold marks the best value in each column.

Fine-tuning moved every metric against the untrained base model on the same 2,700 responses:

	d1-2B, untrained	d1-2B RAG Verifier
F1	70.9	81.5
AUROC	0.877	0.936
Accuracy	82.4%	86.8%
Expected calibration error (10 bins)	0.037	0.030
Answers decided at 95% accuracy	42%	69%
Answers decided at 99% accuracy	9%	16%

Summaries gained the most. The untrained model caught 16.7% of hallucinated summaries; the verifier catches 62.7%.

Usage

Transformers

pip install "transformers>=5.19" torch torchvision pillow

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "landtml/d1-2B-RAG-Verifier", trust_remote_code=True, dtype=torch.bfloat16
).to("cuda")

CHECK = {"supported": {"type": "noul",
                       "instructions": "Is every claim in the answer supported by the source?"}}

source = ("The Riverside branch library reopens on 4 March after a six-month renovation. "
          "Opening hours are 9:00 to 18:00 on weekdays and 10:00 to 14:00 on Saturdays. "
          "The branch is closed on Sundays.")
request = "When is the Riverside library open?"

for answer in ["It is open 9:00 to 18:00 on weekdays and 10:00 to 14:00 on Saturdays.",
               "It is open every day from 9:00 to 18:00, including Sundays."]:
    out = model.system_one({"source": source, "request": request, "answer": answer}, CHECK)
    print(round(out["answers"]["supported"]["noul"], 3), answer)
# 0.987 It is open 9:00 to 18:00 on weekdays and 10:00 to 14:00 on Saturdays.
# 0.004 It is open every day from 9:00 to 18:00, including Sundays.

noul is the probability that the answer is fully supported. Use the exact question text above: the model was trained on it. The state keys source, request and answer are the ones used in training. For data-to-text, RAGTruth puts the structured record (JSON) in source and the writing instruction in request. model.system_one_batch scores many answers in one call.

Choosing a threshold

Flag an answer as unsupported when noul is below 0.56. That threshold was chosen on the development split before the test set was scored, and gives the 81.5 F1 above. Two other ways to use the score:

• Route by confidence. Pass answers above a high cut, block those below a low cut, and send the middle band to a reviewer or a larger model. On the test set, the 69% most confident verdicts are 95% accurate.
• Rank. Sort a batch of answers by noul and review from the bottom.

llama.cpp (GGUF)

gguf/ holds a Q8_0 build for llama.cpp releases with d1 support (PR #31275, late October 2026), tested with build b12173.

llama-server -m d1-2B-RAG-Verifier-Q8_0.gguf --mmproj mmproj-d1-2B-Q8_0.gguf -fa on -c 8192 --port 8080

curl -s http://127.0.0.1:8080/v1/systemone -H "Content-Type: application/json" -d '{
  "state": {"source": "The branch is closed on Sundays.",
            "request": "Is the library open on Sunday?",
            "answer": "Yes, it opens at 10:00 on Sundays."},
  "questions": {"supported": {"type": "noul",
                "instructions": "Is every claim in the answer supported by the source?"}}}'

The Q8_0 file was checked against the bfloat16 weights on the 1,500 development answers: mean absolute difference in noul 0.003, AUROC 0.926 for both, F1 at the 0.56 threshold 83.2 vs 83.3. Median latency on a 12 GB Radeon RX 6750 XT was 144 ms per answer. The GGUF sets the yes/no temperature to 1.0 in its metadata, matching calibration.json.

Files

Path	Contents
model.safetensors	Merged weights, bfloat16. Load with AutoModel as above.
calibration.json	The base model's calibration file with the yes/no temperature set to 1.0. The LoRA was trained on untempered scores, so this keeps the published probabilities identical to the evaluated ones.
adapter/	The LoRA weights alone (PEFT format, r=16, alpha=32, 33 MB) for use on top of LiquidAI/d1-2B.
gguf/	Q8_0 GGUF of the merged model and the unchanged vision projector from the base release.
eval/	Test predictions for this model and for the untrained base, development predictions with their document groups, and the scoring script.
assets/	The charts on this page.

Training

Base model	LiquidAI/d1-2B (bfloat16)
Method	LoRA, r=16, alpha=32, no dropout, on the attention, short-convolution and feed-forward projections of the language model. The vision encoder is untouched.
Objective	Negative log-likelihood of the gold yes/no option for the question above, read from the same slot the model uses at inference
Data	RAGTruth training split, 15,090 responses. Sources longer than 12,000 characters were dropped. 250 source documents (1,500 responses) were held out as a development split; the remaining 13,590 responses were used for training.
Labels	Supported if annotators marked no hallucinated span; 44.7% of training responses are hallucinated
Optimizer	AdamW, learning rate 1e-4, cosine schedule with 5% warmup, about 16 responses per step, gradient clipping at 1.0
Schedule	2 epochs, development split scored every 3,400 responses
Hardware	One RTX 5090, 29 minutes, about 19 responses per second

The released checkpoint is the one with the best development AUROC (0.926), reached at the end of the first epoch. The second epoch did not improve it.

Evaluation protocol

• The development split is grouped by source document, so no development document appears in training. Checkpoint and threshold were both chosen on it.
• The RAGTruth test split was scored once with the selected checkpoint and threshold. It was not used for any choice.
• The untrained d1-2B row was scored on the same 2,700 responses through llama.cpp (Q8_0) with the same question.
• Throughput was measured on the test set itself: 2,700 answers in 66 s on one RTX 5090 (bfloat16, batched by length).
• python eval/evaluate.py eval/test_predictions.jsonl reproduces this model's numbers; python eval/evaluate.py eval/test_predictions_untrained.jsonl 0.5 reproduces the untrained row.

Limitations

• One benchmark. All test numbers come from RAGTruth, whose responses were written in 2023 by six models: GPT-3.5-turbo, GPT-4, Llama-2-chat 7B, 13B and 70B, and Mistral-7B-Instruct. Answers from newer generators, other domains or other languages may score differently. Check it on a sample of your own data before relying on the threshold.
• Response-level only. The model says whether an answer contains an unsupported claim, not where. It does not return spans.
• Context length. Training sources were at most 12,000 characters. Longer documents run, but were not evaluated.
• Possible overlap. The base model's training data is not public, so overlap with RAGTruth cannot be ruled out. The fine-tune gains are measured against the same base model and are not affected by this.
• English. RAGTruth is English. d1-2B supports other languages, but this fine-tune was not evaluated on them.
• Comparison caveat. Third-party rows are copied from their papers, which report response-level results on the RAGTruth test split with their own evaluation code.

License

The verifier is a derivative of LiquidAI/d1-2B and is released under the LFM Open License v1.0. The RAGTruth data used for training is released under the MIT License.

Citation

RAGTruth:

@misc{wu2023ragtruth,
  title         = {RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models},
  author        = {Yuanhao Wu and Juno Zhu and Siliang Xu and Kashun Shum and Cheng Niu and Randy Zhong and Juntong Song and Tong Zhang},
  year          = {2023},
  eprint        = {2401.00396},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL}
}

LettuceDetect (source of the LettuceDetect and RAG-HAT rows): arXiv:2502.17125.