---
tags:
- text faithfulness
- hallucination detection
- RAG evaluation
- cognitive statements
- factual consistency
---


# CogniDet: Cognitive Faithfulness Detector for LLMs

**CogniDet** is a state-of-the-art model for detecting **both factual and cognitive hallucinations** in Large Language Model (LLM) outputs. Developed as part of the [CogniBench](https://github.com/FUTUREEEEEE/CogniBench) framework, it specifically addresses the challenge of evaluating inference-based statements beyond simple fact regurgitation.

## Key Features ✨
1. **Dual Detection Capability**
Identifies both:
- **Factual Hallucinations** (claims contradicting provided context)
- **Cognitive Hallucinations** (unsupported inferences/evaluations)

2. **Legal-Inspired Rigor**
Incorporates a tiered evaluation framework (Rational → Grounded → Unequivocal) inspired by legal evidence standards

3. **Efficient Inference**
Single-pass detection with **8B parameter Llama3 backbone** (faster than NLI-based methods)

4. **Large-Scale Training**
Trained on **CogniBench-L** (24k+ dialogues, 234k+ annotated sentences)

## Performance 🚀
| Detection Type | F1 Score |
|----------------------|----------|
| **Overall** | 70.30 |
| Factual Hallucination| 64.40 |
| **Cognitive Hallucination** | **73.80** |

*Outperforms baselines like SelfCheckGPT (61.1 F1 on cognitive) and RAGTruth (45.3 F1 on factual)*

## Usage 💻
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "future7/CogniDet"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

def detect_hallucinations(context, response):
inputs = tokenizer(
f"CONTEXT: {context}\nRESPONSE: {response}\nHALLUCINATIONS:",
return_tensors="pt"
)
outputs = model.generate(**inputs, max_new_tokens=100)
return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example usage
context = "Moringa trees grow in USDA zones 9-10. Flowering occurs annually in spring."
response = "In cold regions, Moringa can bloom twice yearly if grown indoors."

print(detect_hallucinations(context, response))
# Output: "Bloom frequency claims in cold regions are speculative"
```

## Training Data 🔬
Trained on **CogniBench-L** featuring:
- 7,058 knowledge-grounded dialogues
- 234,164 sentence-level annotations
- Balanced coverage across 15+ domains (Medical, Legal, etc.)
- Auto-labeled via rigorous pipeline (82.2% agreement with humans)

## Limitations ⚠️
1. Best performance on **English** knowledge-grounded dialogues
2. Domain-specific applications (e.g., clinical diagnosis) may require fine-tuning
3. Context window limited to 8K tokens

## Citation 📚
If you use CogniDet, please cite the CogniBench paper:
```bibtex
@inproceedings{tang2025cognibench,
title = {CogniBench: A Legal-inspired Framework for Assessing Cognitive Faithfulness of LLMs},
author = {Tang, Xiaqiang and Li, Jian and Hu, Keyu and Nan, Du
and Li, Xiaolong and Zhang, Xi and Sun, Weigao and Xie, Sihong},
booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)},
year = {2025},
pages = {xxx--xxx}, % 添加页码范围
publisher = {Association for Computational Linguistics},
location = {Vienna, Austria},
url = {https://arxiv.org/abs/2505.20767},
archivePrefix = {arXiv},
eprint = {2505.20767},
primaryClass = {cs.CL}
}
```

## Resources 🔗
- [CogniBench GitHub](https://github.com/FUTUREEEEEE/CogniBench)