README.md

---
license: cc-by-nc-sa-4.0
---

# Model Card for Hallucination Detection Model (HDM-2-3B)

**Paper:** 
[![Read full-text on arXiv](https://img.shields.io/badge/arXiv-2504.07069-b31b1b.svg)](https://arxiv.org/abs/2504.07069)
*HalluciNot: Hallucination Detection Through Context and Common Knowledge Verification.*


**Notebook:** [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https\://colab.research.google.com/drive/1HclyB06twZVIxuK6AlyifRaf77vO5Yz#scrollTo=UVvBvBMWrDiv)


**HDM-Bench Dataset:** 🤗 [HuggingFace Dataset](https\://huggingface.co/datasets/AimonLabs/HDM-Bench)


## Introduction

Most judge models used in the industry today are not specialized for Hallucination evaluation tasks. 
Developers using them often struggle with score inconsistency, variance, high latencies, high costs, and prompt sensitivity. 
HDM-2 solves these challenges and at the same time, provides industry-first, state-of-the-art features.


## Highlights:

- Outperforms existing baselines on RagTruth, TruthfulQA, and our new HDM-Bench benchmark.

- **Context-based** hallucination evaluations based on user-provided or retrieved documents.

- **Common knowledge** contradictions based on widely-accepted common knowledge facts.

- **Phrase, token, and sentence-level** Hallucination identification with token-level probability **scores**

- Generalized model that works well across a variety of domains such as Finance, Healthcare, Legal, and Insurance.

- Operates within a **latency** budget of **500ms** on a single L4 GPU, especially beneficial for Agentic use cases.

## Model Overview:

HDM-2 is a modular, production-ready, multi-task hallucination (or inaccuracy) evaluation model designed to validate the factual groundedness of LLM outputs in enterprise environments, for both **contextual** and **common knowledge** evaluations. 
HDM-2 introduces a novel taxonomy-guided, span-level validation architecture focused on precision, explainability, and adaptability. 
The figure below shows the workflow (on the left) in which we determine whether a certain LLM response is hallucinated or not and an example (on the right) that shows the taxonomy of an LLM response.

HDM-2 Model Workflow | Example of Enterprise LLM Response Taxonomy
--- | ---
![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXdpn0qSjx_A3ax0qXZ3BIBTXAbMphuN1gLPXRQ4m_aTCSaN_hMMS27d0hJeQaZhc0P_iCpnktRsCyT_xB5V7-ofqQwjAvNWkRka_fJAGKfD466PK-jgGoRpDPqT9Ag3MT8XVSGscQ?key=x9HqmDQsJmBeqyuiakDxe8Cs) | ![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXfJzyMnYVlR9sNIV7cDKmY3d_RnQYUBj7Ass6RWfhTt5ds2OJ5os2uPv7loECI_ao7_To3H4WV9UoHhnbJ2Ux-XSFQK76NJzOkiWNuDQQxuaojzgazujJ45KPSyhbtbfNe3msyl6w?key=x9HqmDQsJmBeqyuiakDxe8Cs)


### Enterprise Models

- The Enterprise version offers a way to incorporate “Enterprise knowledge” into Hallucination evaluations. This means knowledge that is specific to your company (or domain or industry) that might not be present in your context!!

- Another important feature covered in the Enterprise version are explanations. Please reach out to us for Enterprise licensing. 

- Other premium capabilities that will be included in the Enterprise version include improved accuracies, even lower latencies, and additional use cases such as Math and Code.

- Apart from Hallucinations, we have SOTA models for Prompt/Instruction adherence, RAG Relevance, Reranking (Promptable). The instruction adherence model is general-purpose and extremely low-latency. It performs well with a wide variety of instructions, including safety, style, and format constraints.


### Performance - Model Accuracy

See paper (linked on top) for more details.

|             |               |            |              |
| :---------: | :-----------: | :--------: | :----------: |
| **Dataset** | **Precision** | **Recall** | **F1 Score** |
|   HDMBENCH  |      0.87     |    0.84    |     0.855    |
|  TruthfulQA |      0.82     |    0.78    |     0.80     |
|   RagTruth  |      0.85     |    0.81    |     0.83     |


### Latency


|                         |                      |                        |                         |                     |
| ----------------------- | -------------------- | ---------------------- | ----------------------- | ------------------- |
| **Device**              | **Avg. Latency (s)** | **Median Latency (s)** | **95th Percentile (s)** | **Max Latency (s)** |
| Nvidia A100             | 0.204                | 0.201                  | 0.208                   | 1.32                |
| Nvidia L4 (recommended) | 0.207                | 0.203                  | 0.220                   | 1.29                |
| Nvidia T4               | 0.935                | 0.947                  | 1.487                   | 1.605               |
| CPU                     | 261.92               | 242.76                 | 350.76                  | 356.96              |



## How to Get Started with the Model

Use the code below to get started with the model.

Install the Inference Code

    !pip install hdm2 --quiet

Run the HDM-2 model

    # Load the model from HuggingFace into the GPU

<!---->

    from hdm2 import HallucinationDetectionModel

<!---->

    hdm_model = HallucinationDetectionModel()

<!---->

    prompt = "Explain how the heart functions"
    context = """
    The heart is a muscular organ that pumps blood throughout the body.
    It has four chambers: two atria and two ventricles.
    """
    response = """The heart is a vital six-chambered organ that pumps blood throughout the human body.
    It contains three atria and three ventricles that work in harmony to circulate blood.
    The heart primarily runs on glucose for energy and typically beats at a rate of 20-30 beats per minute in adults.
    Located in the center-left of the chest, the heart is protected by the ribcage.
    The average human heart weighs about 5 pounds and will beat approximately 2 million times in a lifetime.
    """
    # Ground truth:
    # Hearts have 4 chambers (not 6), have 2 atria and 2 ventricles (not 3 each),
    # normal heart rate is 60-100 BPM (not 20-30),
    # average heart weighs ~10 oz (not 5 pounds),
    # and beats ~2.5 billion times (not 2 million) in a lifetime

<!---->

    # Detect hallucinations with default parameters
    results = hdm_model.apply(prompt, context, response)

Print the results

    # Utility function to help with printing the model output
    def print_results(results):
     #print(results)
     # Print results
     print(f"\nHallucination severity: {results['adjusted_hallucination_severity']:.4f}")

<!---->

     # Print hallucinated sentences
     if results['candidate_sentences']:
         print("\nPotentially hallucinated sentences:")
         is_ck_hallucinated = False
         for sentence_result in results['ck_results']:
             if sentence_result['prediction'] == 1:  # 1 indicates hallucination
                 print(f"- {sentence_result['text']} (Probability: {sentence_result['hallucination_probability']:.4f})")
                 is_ck_hallucinated = True
         if not is_ck_hallucinated:
           print("No hallucinated sentences detected.")
     else:
         print("\nNo hallucinated sentences detected.")

<!---->

    print_results(results)


### Model Description

- Model ID: HDM-2-3B

- Developed by: AIMon Labs, Inc.

- Model type: 

- Language(s) (NLP): English

- License: CC BY-NC-SA 4.0

- License URL: <https://creativecommons.org/licenses/by-nc-sa/4.0/> 


### Model Sources

- Code repository: [GitHub](https://github.com/aimonlabs/hallucination-detection-model)

- Model weights: [HuggingFace](https://huggingface.co/AimonLabs/hallucination-detection-model/)

- Paper: [arXiv](https://arxiv.org/abs/2504.07069)

- Demo: [Google Colab](https://colab.research.google.com/drive/1HclyB06twZVIxuK6AlyifRaf77vO5Yz#scrollTo=UVvBvBMWrDiv)


## Uses

### Direct Use

1. Automating Hallucination or Inaccuracy Evaluations

2. Assisting humans evaluating LLM responses for Hallucinations

3. Phrase, word or sentence-level identification of where Hallucinations lie

4. Selecting the best LLM with the least hallucinations for specific use cases

5. Automatic re-prompting for better LLM responses


## Limitations

- Annotations of "common knowledge" may still contain subjective judgments


## Technical Specifications 

See paper for [more details](https://arxiv.org/abs/2504.07069)


## Citation:

@misc {hdm-2-3b,

    author       = {Paudel, A. and Lyzhov, A. and AIMon Labs},

    title        = {HDM-2-3B: Hallucination Detection Model for Enterprise LLMs}, [Bibek Paudel](mailto:bibek@aimon.ai) 

    year         = 2025,

    url          = {<https://huggingface.co/aimonlabs/> ??????? [Preetam Joshi](mailto:preetam@aimon.ai)},

    publisher    = {AIMon Labs, Inc.},

    eprint       = {2504.07069},

    archivePrefix= {arXiv},

    primaryClass = {cs.CL},

    url          = {https\://arxiv.org/abs/2504.07069}

}


## Model Card Authors

@bibekp, @alexlyzhov-aimon, @pjoshi30, @aimonp


## Model Card Contact

<info@aimon.ai>, @aimonp, @pjoshi30