Hallucination Detection via Internal Representations: An Empirical Study

---

Abstract. — This report summarizes a series of experiments replicating and extending prior work on extracting hallucination-related signals from an LLM's internal representations. Using Ministral-8B-Instruct-2410 as a case study, we constructed a dataset of 145,169 labeled examples spanning diverse domains and trained lightweight XGBoost probes on mean-pooled hidden states and attention features. Results confirm that hallucination-relevant information is encoded in the model's activations, achieving approximately 84% accuracy, 91% AUC, and 75% F1 score. Notably, this signal is concentrated in specific mid-layers rather than distributed across the network, with layer 16 (of 36) proving most informative. Multi-layer concatenation and attention-based features yielded no substantial improvement over the single best-performing layer.

The full experimental pipeline is documented in a series of four notebooks.

---

1. Introduction

Large language models occasionally generate plausible-sounding but factually incorrect content, a phenomenon commonly referred to as "hallucination". Current detection methods typically rely on external knowledge retrieval, self-consistency checks, or secondary judge models, each incurring significant computational overhead or introducing additional points of failure.

An alternative approach, explored in recent work, is to extract hallucination-related signals directly from the model's internal representations. Azaria & Mitchell (2023) demonstrated that classifiers trained on hidden states can detect statement truthfulness with 71-83% accuracy. Ji et al. (2024) extended this to pre-generation risk estimation, while other work has explored eigenvalue-based consistency metrics (Chen et al., 2024) and attention-based probing (Sriramanan et al., 2024).

This study replicates and extends these findings using a different model, a larger dataset, and alternative methodological choices. Specifically, we use mean-pooled activations (rather than single-token extraction), XGBoost classifiers (rather than MLP probes), and a dataset of 145K examples spanning multiple domains.

We focus on Mistral's Ministral-8B-Instruct-2410, a 36-layer transformer with 32 attention heads and a hidden dimension of 4096. This model was chosen for its moderate size, facilitating rapid experimentation and activation extraction on a single NVIDIA A100 GPU.

2. Dataset

A dedicated dataset was constructed for this research, now publicly available at krogoldAI/hallucination-labeled-dataset. The dataset comprises 145,169 labeled examples specifically generated by Ministral-8B-Instruct-2410 at temperature 0.7, with an overall hallucination rate of 32.2%.

2.1. Construction

The dataset aggregates prompts from six source datasets spanning diverse domains and tasks. For each prompt, Ministral-8B-Instruct-2410 generated a response, which was then labeled for hallucination presence. Labeling was performed via judge LLM evaluation, except for one source dataset where deterministic comparison with ground truth answers was possible. Full details on source datasets as well as system prompts used for evaluation are provided in the dataset card.

2.2. Design Considerations

Two deliberate choices shaped the dataset construction:

1. Same-model generation: All outputs were produced by the same model used for probing (Ministral-8B-Instruct-2410). This avoids confounding effects from out-of-distribution text, which could manifest as elevated perplexity and atypical activation patterns.

2. Balanced splits: The data is partitioned 70/15/15 into train, validation, and test sets, preserving per-source distribution and hallucination rates across splits to ensure consistent evaluation conditions.

3. Methodology

3.1. Feature Extraction

Activations were extracted using llmscan, a lightweight library developed for this research. For each example, the model processed the full conversation (user prompt and assistant response) formatted according to Mistral's chat template. Crucially, only tokens corresponding to the assistant's response (following the [/INST] delimiter) were retained for feature extraction, ensuring the probe operates on the model's representation of its own output (where hallucinations manifest).

Two types of features were extracted:

1. Hidden state activations: The output of the feed-forward network (FFN) at each layer, mean-pooled across assistant response tokens, yielding a 4096-dimensional vector per layer.

2. Attention features: For selected layers, we extracted both the multi-head attention (MHA) output (4096 dimensions) and per-head statistical summaries. For each of the 32 attention heads, 12 statistics were computed over the attention distribution:

   • Distributional measures: entropy, standard deviation, Gini coefficient, Simpson index
   • Concentration metrics: maximum attention weight, top-5 cumulative mass, top-10% cumulative mass, effective token count
   • Positional statistics: mean relative distance (average position attended to), attention mass on the beginning-of-sequence token
   • Shape measure: skewness
   • Global measure: Frobenius norm (capturing overall "energy" of the attention pattern)

   This yields 384 attention statistics per layer (12 metrics × 32 heads).

3.2. Probe Architecture

All probes used XGBoost classifiers with consistent hyperparameters: 800 estimators, max depth 6, learning rate 0.05, 80% subsampling, and GPU-accelerated training. Early stopping (patience of 20 rounds) prevented overfitting.

3.3. Experimental Protocol

Experiments proceeded in four stages:

1. Layer selection: Train a simple probe on each layer's activations using validation data; rank layers by accuracy and F1 to identify the most expressive candidates.

2. Single-layer probing: Train a full probe on the best-performing layer using the complete training set.

3. Multi-layer probing: Concatenate activations from multiple top-performing layers; evaluate with and without feature selection.

4. Attention-based probing: Train probes on attention statistics and MHA outputs; evaluate hybrid approaches combining attention and activation features.

4. Results

4.1. Layer Selection

The layer selection analysis revealed that mid-layers are most expressive for hallucination detection. Layers 16, 15, 18, and 21 consistently outperformed both early and late layers.

Figure 1: Layer-wise probe performance (F1 and accuracy) across all 36 layers of Ministral-8B-Instruct-2410.

This pattern admits an intuitive interpretation: early layers (1-10) encode primarily syntactic and surface-level features, while late layers (25-36) may be optimized for generation rather than factual grounding. Mid-layers appear to capture semantic representations where hallucination-relevant signals are most accessible.

Remark: This mid-layer concentration aligns with Azaria & Mitchell (2023), who also found middle layers most predictive when probing true/false statements. Note, however, that Ji et al. (2024) reported deeper layers as optimal when probing queries to predict hallucination risk before generation. This discrepancy may reflect a genuine difference between pre-generation uncertainty signals (encoded in query representations) and post-generation reliability signals (encoded in response representations).

4.2. Single-Layer Results

Training on layer 16 alone yielded the following results:

Threshold optimization improves hallucination recall from 65% to 76% at modest cost to precision, which is a reasonable tradeoff given that missing hallucinations is typically more costly than occasional false positives. In any event, this confirms that hidden states encode, at least partially, hallucination-related signals.

4.3. Multi-Layer Results

Concatenating multiple layers yielded marginal improvements:

Performance plateaus regardless of whether one, two, or three layers are used, with all configurations converging to approximately 74-75% F1, 83-84% accuracy, and 91% AUC. Feature selection reveals substantial redundancy across layer representations, suggesting that hallucination-related signals are concentrated rather than distributed. Overall, reducing dimensionality does not improve performance.

4.4. Attention-Based Results

Attention features underperformed hidden state activations:

Hybrid approaches combining activations with attention features showed no improvement over activations alone:

5. Discussion

5.1. Key Findings

Our results confirm that hallucination-related signals are present and extractable from the model's internal representations, consistent with prior work. The consistent 91% AUC across configurations demonstrates that this information can be retrieved using lightweight probes, without requiring external knowledge bases or additional inference passes.

Equally notable is that these signals seem to be "localized", not distributed. Indeed, performance does not improve with additional layers: the hallucination-related signal appears concentrated in specific mid-layers (particularly layer 16) rather than spread across the network. This concentration has practical implications: a single layer's activations suffice for detection, minimizing extraction overhead.

Moreover, attention features - both per-head statistics and MHA outputs - added no value beyond what FFN activations already provided. The discriminative information appears to reside primarily in the feed-forward pathway, suggesting that attention patterns, at least as summarized by our chosen statistics, do not capture complementary hallucination-relevant signals.

Finally, all configurations converge to similar performance bounds: approximately 75% F1, 83-84% accuracy, and 91% AUC. These results are comparable to prior work (Azaria & Mitchell report 71-83% accuracy; Ji et al. report 84.32%), suggesting we are approaching the practical ceiling for this general approach. This ceiling may reflect fundamental limits of mean-pooled representations, noise in dataset labels, or inherent ambiguity in what constitutes a "hallucination".

5.2. Limitations and Future Directions

Several factors may account for the observed performance ceiling:

• Pooling artifacts: Mean-pooling may attenuate per-token signals, particularly for longer responses where averaging "dilutes" token-level information. (The non-negligible proportion of short assistant responses in the dataset, where pooling has less effect, may, in fact, partly contribute to the positive results observed.) Retaining raw token-level activations and training a sequence-aware probe (e.g., a small transformer) could capture finer-grained patterns, though at the cost of the lightweight inference we sought to preserve.

• Dataset quality: Due to budget constraints, labels were assigned by judge LLMs without majority voting or human verification. While this enabled large-scale dataset construction, a non-negligible false positive/negative rate may impose an upper bound on achievable performance regardless of probe sophistication.

• Fundamental limits: The hallucination signal encoded in activations may be inherently partial or may not align perfectly with standard definitions of factual hallucination. In particular, the probe might be detecting something akin to the model committing to an answer despite low internal confidence, rather than factual incorrectness per se. Such low-confidence commitments sometimes produce correct answers and sometimes incorrect ones, yet correctness-based hallucination labels treat these internally similar states differently. This mismatch could impose a fundamental ceiling on probe performance.

Additional approaches explored but not detailed here - including alternative pooling strategies (max-pooling, attention-weighted pooling), alternative classifiers (random forests, logistic regression, neural probes), probe ensembling, and PCA dimensionality reduction - yielded no improvements. Mean-pooling with XGBoost consistently performed best.

Two directions could help clarify the source of the observed performance ceiling. First, dataset refinement through human-verified labels or robust majority voting would establish whether label noise is the limiting factor. Second, exploring sequence-aware probes that operate on raw token-level activations would test whether mean-pooling discards discriminative per-token signals. However, both directions involve significant tradeoffs: human annotation and majority voting multiply labeling costs, while retaining full activation sequences introduces substantial storage and computational overhead.

6. Conclusion

This study confirms that hallucination-related signals are present and extractable from transformer internal representations using lightweight classical probes, replicating prior findings at larger scale and with alternative methodological choices. A single mid-layer's mean-pooled activations suffice to achieve 84% accuracy, 91% AUC, and 75% F1 score. While the observed performance ceiling leaves room for improvement, these findings reinforce the viability of real-time, low-overhead hallucination detection and offer additional data points on how factual grounding may be represented within language models.

References

• Azaria, A., & Mitchell, T. (2023). The Internal State of an LLM Knows When It's Lying. Findings of EMNLP 2023.
• Chen, C., et al. (2024). INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection. ICLR 2024.
• Ji, Z., et al. (2024). LLM Internal States Reveal Hallucination Risk Faced With a Query. BlackboxNLP Workshop 2024.
• Sriramanan, G., et al. (2024). LLM-Check: Investigating Detection of Hallucinations in Large Language Models. NeurIPS 2024.

Resources

• Dataset: krogoldAI/hallucination-labeled-dataset
• Code: llmscan library
• Model: Ministral-8B-Instruct-2410