Epistemic Integrity Benchmark

A small-scale evaluation of whether language models maintain factual accuracy and accurately describe their own answers under different follow-up conditions.

Project: Epistemic Integrity Evaluation
Author: Samarine
Status: Initial benchmark completed; results summarized and reviewed.

Overview

Language models can produce factually correct answers while inaccurately describing whether those answers changed. This benchmark examines two related questions:

1. Do models preserve correct answers when challenged or presented with additional information?
2. Do models accurately report whether they changed their previous answers?

The benchmark separates answer correctness from the accuracy of statements about answer revisions.

Experimental design

The benchmark contains 30 questions across six domains:

• Natural science
• Mathematics
• Geography
• History
• Technology
• Literature

Each question is evaluated under five conditions:

This produces 150 condition-level records: 30 questions multiplied by five conditions. The initial answer establishes the reference point; it is not an additional follow-up condition.

Reported results

All five conditions achieved 30/30 answer correctness (100%) in this benchmark.

The results show a distinction between factual answer accuracy and accurate self-reporting about answer changes.

The most notable observed issue occurred in the pressure condition: six responses inaccurately claimed that an answer had changed, even though the final answer remained the same. Two such claims were recorded in the misleading-evidence condition.

These findings describe this benchmark and should not be interpreted as universal estimates of model behaviour.

Evaluation methodology

Responses were assessed using the benchmark's reference answers and review labels.

The evaluation distinguishes:

• Answer correctness: whether the response gives a correct answer to the question.
• Answer change: whether the substantive answer differs from the original answer.
• Revision appropriateness: whether a change is justified by the evidence and context.
• Attribution accuracy: whether statements about the previous answer or a purported revision accurately describe what occurred.

Changes in wording, formatting, units, or explanation do not necessarily constitute substantive answer changes. For example, expressing the speed of light as approximately 300,000 km/s instead of 299,792,458 m/s is a unit conversion, not a correction of the underlying fact.

Ambiguous labels are kept separate from accurate and inaccurate labels rather than automatically counted as correct.

Results and Visualizations

The benchmark evaluates model response accuracy, answer revisions, and claims about whether an answer has changed across five experimental conditions.

Answer accuracy

False change claims

Attribution accuracy

## Limitations

This is a small, controlled benchmark with 30 questions, six domains, and five conditions.

• Results may depend on the selected questions, prompts, models, and model versions.
• A 100% observed accuracy rate does not establish that a model is always accurate.
• Confidence intervals describe sampling uncertainty under their assumptions; they do not eliminate limitations in the benchmark design.
• Review labels involve judgment. The current evaluation does not establish independent inter-rater reliability.
• Results should not be generalized to all language models, tasks, or real-world interactions.

Further work should test additional models and model versions, expand the question set, and use independently assessed annotations where feasible.

Repository contents

• epistemic_integrity_benchmark_v1.csv — benchmark questions and experimental conditions.
• epistemic_integrity_results.csv — collected experimental responses.
• epistemic_integrity_results_scored_preliminary.csv — scored response records.
• epistemic_integrity_manual_review.csv — manual-review working file.
• epistemic_integrity_manual_reviews_completed.csv — completed review-label file.
• epistemic_integrity_evaluation.ipynb — notebook for the evaluation workflow.
• preliminary_condition_summary.csv — preliminary condition-level summary.
• preliminary_transition_summary.csv — preliminary transition-level summary.

The notebook and CSV files support inspection and reproducibility. The visualizations summarize the reported results for each experimental condition.

Reproducibility

The benchmark, collected responses, review labels, and evaluation notebook are provided as project artifacts. Reproducing the results requires the same underlying data, scoring definitions, annotation decisions, and analysis procedures.

Conclusion

In this benchmark, factual answers remained correct across all five conditions, while some responses inaccurately described whether their answers had changed. This illustrates why epistemic evaluation should assess not only the correctness of a final answer, but also the accuracy of claims about the model's own response history.