Evidence Composition Fidelity: A Blindspot in How LLMs Compose Evidence

1. The Blindspot

Part of my job at the Nigeria Revenue Service (the Service), the apex tax agency in Nigeria, aside from heading my unit, is to build and deploy AI systems and agents used in the work we do in our unit and in the Service at large. However, a recurring problem I have come across in my work and unable to concretely solve for so far is that the systems/agents we have built find it hard to correctly compose evidence. For context, we normally work with streams of evidence, which includes filings, reports, correspondence, etc from companies. We pass these streams of evidence to the AI and then ask it to reason over that stream, not just answer a fixed factual question. In that setting, the main pain points/blind spots we keep coming across over and over again, regardless of which LLM model or agent framework we use are:

• Order robustness is taken lightly: we noticed that more recent data always tends to override older data by default, even when the older data is not wrong or specifically superseded. The data has just been said recently, or last, in the conversation compared to the other data in the same conversation. The AI at most times happily lets the last thing it read/saw dominate its answer without caring whether or not that is the right thing to weight most heavily in the conversation.
• Dependence discounting is basically absent: in our use cases, a company might report the same figure in three or more different places (e.g. in a filing, a cover letter, a supporting schedule, an email correspondence, etc) and the AI system most often times than not treats the many different reports of the same figure as many confirmations (from independent sources) when it is really one number, repeated in different places by the same entity (in our case same company). Real independent corroboration and mere repetition get treated the same way by most AI models/agent frameworks.
• Temporal override is unreliable: in some cases, a company files a correction. For example, this year's numbers supersedes the numbers that were reported last year or the initial estimate is later revised. An agent should update its working belief and take the correction to account by correctly overriding the old numbers with the new numbers. However, this is sadly not the case. As we have noticed many times, corrections often get lost, especially once everything else is discussed in-between; the model can (and usually does) drift back toward the earlier, wrong number that has been overridden.
• Numerical fidelity breaks down under pressure: given the sector we work in (taxation and finance), an exact figure like ₦4,189,204.20 must, at all times stay exactly ₦4,189,204.20, especially for tax purposes, if we are to get the right reports and risk analyses out. But models often round it, "clean it up", or drift toward a rounder or more typical-looking number. In some worse cases, when a value is missing or redacted in the company's document, a model will sometimes just invent a number that is plausible in its place instead of flagging it or mentioning it as missing in its answer, which is close to the worst possible failure mode in a compliance context.

The above blindspot has made the systems we have built non-trustworthy and undeployable, at least not without human oversight, given that compliance failure could lead to wrongful taxation, which in turn could lock the Service in unending litigations, which in turn would lead to loss of revenue to the federal government.

Sadly, none of the above blindspot evaluations show up in standard LLM benchmarks, which mostly test whether a model knows a fact it was trained on or given, i.e. a fixed, single-shot lookup. The blindspot I am describing is different. It is about how a model composes a live stream of evidence into a correct belief, as that evidence arrives, especially when that evidence may update and sometimes contradict itself or existing evidence, which is exactly what happens in a real tax administration workflow (and perhaps, I'd argue, many other fields including physical sciences, finance, sports, legal review, medical use cases, journalism, etc). I call this evidence composition fidelity, and this evaluation is built to test it directly.

2. Evaluating the gap

Model: Qwen/Qwen2.5-1.5B-Instruct (1.5B parameters, within the 0.6–6B range), loaded and run in Google Colab on a free T4 GPU. I chose this model because it is a small instruction-tuned model from a family of models whose larger versions are in active use and frontier research. I believe that if a composition failure shows up even at this scale in a model family that is well-regarded, it will be reasonable to bet that the same underlying tendency is present, maybe harder to elicit, in the larger models people actually deploy. My claim is about the shape of the failure. I am not saying that a 1.5B model's exact numbers transfer unchanged to a 70B version.

Design: I built a synthetic dataset (generate_dataset.py or the 4th cell of the work notebook, seeded for reproducibility) around four batteries, one per blind spot discussed above:

• Order robustness battery: we generate this battery using a fictional entity (e.g. Corvantex-OR7) which gets one confirming and one disconfirming piece of evidence of designed-comparable strength. I ask for a confidence score after hearing them in one order, then run an independent copy of the same conversation with the order reversed, and then I compare the confidence scores.
• Dependence discounting battery: the same fictional entity as above gets either one source, three independent confirming sources, or three dependent sources, i.e., the text of the sources makes it clear that the second and the third are just repeating the first.
• Temporal override battery: in this battery, a claim is reported true in 2019, then a report in 2023 overturns that of 2019. The report in 2023 is unambiguous with the overturning. For this, I ask for confidence "as of 2019" and "right now", sometimes I insert an irrelevant sentence between the correction and the question to test whether the correction survives being crowded by something unrelated.
• Numerical fidelity battery: in this case, real, exact values across three fields (Chemistry, Physics and Finance), in four categories are generated. The Categories are as follows:
  1. an ordinary control value;
  2. a real-but-unusual value near the famous constant (evaluating the question does the model drift toward the constant?);
  3. a value with many decimal places (evaluating the question does it get truncated?);
  4. and a value replaced with an explicit missing-data placeholder (evaluating the question does the model invent a number or does it say the number is missing?).

My choice of fictional entity names for the first three batteries (I also call them the epistemic batteries) above is because I believe this makes the evaluation more truthful as the model will have no pretrained prior about such fictional entities. Therefore, any change in the model's answer has to come from how it processed the evidence I gave it in that specific conversation, not from some prior belief it already holds due to its trained parameters. In contrast, I use real, well-known constants and values in the numeric battery since the effect I am trying to evaluate there (i.e. drift towards a "typical" number) only exists if such a typical value actually exists in the model's training data.

The structure of the evaluation is such that every battery was run under a 2x2 design, which are:

• 2 prompt styles (a plain instruction [weak], vs. an explicit "don't round, don't blend, don't invent" prompt rider [strict]) X
• 2 context textures (the model sees one piece of evidence in isolation [isolated], vs. embedded in a fuller stream of evidence [sequential]).

This was the most important methodological decision I made in this whole study because it lets me tell the difference between "a shallow prompting problem that a better and stricter system prompt fixes" and "a deeper compositional failure that survives even after being explicitly told not to happen explicitly". I ran 15 items/pairs per condition for the three epistemic batteries and 8 numeric variants across 3 domains per condition. This makes a total of 816 model generations in total. I then computed 95% bootstrap confidence intervals on every headline result value, so that I could tell which differences were actually or likely real and not just sampling noise at this sample size.

What I found

1. Order robustness gets worse, not better, under the strict instruction, when evidence is isolated: Across all the four conditions the model shows a bias for recent data. It leans towards whichever evidence piece it heard last. But the largest swing of any condition (64.9 points of confidence, 95% CI: 51.3–78.3) shows up under the strict prompt with isolated evidence, not the weak/sequential default. The interval doesn't overlap the best performing condition, which is the weak+sequential (24.0, CI 6.0–44.0) at all. The fact that we tell the model explicitly to reason only from the evidence given to it did not fix this. If anything, it correlated with the worst outcome I measured. The fact that the naive fix (a better and stricter instruction) does not touch this failure mode is the single most surprising and, I believe, most important result in this battery.
2. Temporal corrections are fragile, and this fragility is specific to strict prompting: under the strict prompt, especially the case whereby we add one content-free, irrelevant sentence between an unambiguous correction and the question, we notice that the model's accuracy at getting the direction right drops measurably (e.g. strict+sequential: 0.73 direct → 0.33 with distractor, CIs barely overlapping). Under the weak prompt, the effect is noisier and in one condition, it even reverses. This maps directly onto the NRS scenario that motivated this study, that is, a filed correction should not be dislodged by whatever unrelated thing happens to come up next in a case file, and this small model shows exactly that vulnerability.
3. The model invents specific values for data explicitly flagged as missing, at a rate around one-third to two-fifths, in every condition tested: when a value is replaced with an explicit placeholder like [ERROR_DATA_UNAVAILABLE], the model still, roughly a third of the time, states a specific number instead of reporting that the data is missing (rates from 0.333 to 0.417 across all four conditions, all confidence intervals clearly above zero). This is the most direct synthetic analogue of the exact behavior I described from my own work, which is when a model quietly fills a plausible-looking figure where the real one should have been reported as or flagged absent.
4. Dependence discounting revealed two different failures hiding behind one number. Under the strict prompt with isolated evidence, checking gain_independent and gain_dependent separately shows the negative discounting gap is not from over-trusting copies. Confidence from three copies of one source stays roughly flat (gain_dependent, 95% CI −10.0 to 0.0). The negative gap comes from gain_independent itself being reliably negative (95% CI −30.3 to −3.3): seeing three genuinely independent confirming sources under a strict instruction actually lowers the model's confidence below its one-source baseline. Under the weak prompt, both gain_independent and gain_dependent are reliably positive in every condition tested (all four confidence intervals exclude zero), so the model is not simply failing to react to evidence there, as I first hypothesized. It shows a real, if small and only partially reliable, edge toward weighting independent sources more than copies. The strict+sequential condition remains inconclusive on which component drives its gap, since both intervals cross zero at this sample size.
5. Numerical drift and truncation were rarer and less conclusive at this sample size: Cat2 canonical-value drift and Cat3 decimal truncation each appeared in only one of four conditions, on a small base of 24 items, with confidence intervals crossing zero. This battery is worth a much larger numeric sample before I can draw a firm conclusion in it, unlike the above three findings.

Important Note on Methodology

The model actually frequently deviated from the exact requested output format (e.g. writing "Confidence: 75" instead of the literally requested "CONFIDENCE: 75"), especially under the weak prompt. However, I made parsing the model output tolerant of these deviations, since a strict parser would have silently and non-randomly dropped weak prompt responses, thus biasing the whole comparison, which was one of our main goals. For the numeric battery specifically, the model frequently wrote a full narrative report and never reached the required footer restating its values; when that happened, I gave it one follow-up turn with its own reply pre-filled as REPORTED_VALUE[CATn]:, thus forcing the format while leaving the value entirely up to the model to decide (generate). Every row is flagged as parsed on first try or rescued, and both rates are reported in full in results/summary_metrics.json. This is important so nothing is hidden in the numbers reported in this repo.

3. Proposed path forward (speculative)

The failures discussed above come from asking one ordinary LLM to play every role at once. Specifically, the model played the roles of reading evidence, judging its reliability, tracking when it arrived and whether it's a duplicate, and composing a final belief informally, inside a single context window, with no explicit, persistent representation of "what I currently believe" that exists outside that one conversation. My proposed direction, which I'd want to further test include, but not limited to the following:

1. Decompose composition into an explicit pipeline: Instead of one model call doing everything, I propose to test different models handling separate steps. For example, I can dedicate a model to be an extractor that tags each piece of evidence with whether it supports or contradicts a claim, whether it's independent or a copy of something already seen, and when it arrived; and another model that is in charge of a distinct composition step that combines those tags into an updated belief. My guess is that even a small, fixed formula for combining well-tagged evidence would outperform one model doing extraction and composition together and that a learned composition step, trained to be robust to noisy extraction, would do better still.
2. Give the model (or system) a persistent, external belief state: currently, I am making the model re-derive its position from the full raw conversation every time but I am proposing a separate external belief state which the model retrieves from and acts on by updating. In a tax-agency setting this looks like a running case record (current belief, with a provenance trail) that gets updated incrementally as new filings arrive which is usually what is done. As opposed to re-reading the entire case file from scratch on every question, which only happens in very rare cases. My hypothesis is that fidelity should degrade more slowly as the case file grows if belief lives in a compact, explicitly-updated record instead of being re-derived from raw context each time.
3. Train specifically on redundancy and correction, not just single-document QA: this is a concrete, practical fine-tuning idea. I propose to build a data curation strategy around explicitly labeled examples of "these two documents say the same thing, don't double-count it" and "this document supersedes that one, don't drift back." Current instruction-tuning data likely has very little of this, since most QA and instruction data is built around single, static documents rather than corrected or duplicated streams, which would explain why this gap is invisible to standard benchmarks in the first place.
4. On numerical fidelity specifically, I'd want to test whether simply training a model to refuse to state a number it wasn't explicitly given (a stronger, more calibrated "I don't know" for the missing-data case) closes most of the imputation gap, since it doesn't obviously require touching the composition pipeline at all. This might be a much cheaper, narrower fix than the architectural changes above.
5. Build a much larger, open benchmark from this pilot's design, and publish it. This pilot's four batteries (order, dependence, temporal, numerical fidelity) are a narrow, single-model test of a bigger idea: a controlled benchmark where each way evidence composition can go wrong is varied on its own, with known ground truth, so results are not confounded by testing several things at once. A natural next step is extending this design along axes this pilot does not yet cover, for example how a model handles a genuinely novel entity against one it has already seen, how much fidelity survives when the model has to compress or summarize its own history under a fixed memory budget, and evidence that arrives as more than text, for example a scanned report alongside a written correction. I would also want to run this across several models, not just one, and publish the resulting dataset and evaluation code as an open benchmark other researchers can build on and compare against.

None of the proposed paths forward above is proven here. They are all speculative. However, this evaluation shows that the failures exist and persist under an explicit strict instruction; it does not test whether any of the fixes above would actually work. That's the next, larger piece of work.

Metrics glossary

Plain-language definitions of every term used above and in the tables, in the order they first appear.

• Confidence score - a 0-100 number the model states for how strongly it believes a claim, extracted from its response text.
• Prompt style (weak / strict) - weak is a plain, generic instruction ("write a summary of this data"); strict adds an explicit zero-tolerance rider not to round, blend, or invent. Crossing both against context texture is what lets a result be read as "fixed by a better instruction" vs. "survives an explicit instruction not to happen."
• Context texture (isolated / sequential) - isolated means the model sees only the one piece of evidence relevant to the current question; sequential means that evidence is embedded in a fuller, sometimes cluttered stream of other material.
• abs_diff (order robustness) - the absolute difference in stated confidence between hearing the same confirming and disconfirming evidence in one order vs. the reverse. 0 = perfectly order-invariant; higher = more sensitive to order alone.
• signed_diff - the same difference, keeping the sign. Consistently negative means the model leans toward whichever evidence it heard last (recency bias); consistently positive means it leans toward whichever it heard first (primacy bias).
• Cold baseline / zero-evidence prior - the model's stated confidence in a claim with no supporting evidence at all. Used both as a sanity check (a fictional entity should score near 50, i.e. a coin flip) and as the normative target for the order-robustness Brier score.
• Brier score - (stated probability - target probability)^2, averaged over items. 0 is a perfect score; higher is worse. It scores a stated confidence against a specific target, rather than only comparing two conditions to each other.
• Order-robustness Brier target - each entity's own measured cold-baseline confidence, since one confirming and one disconfirming piece of evidence of comparable strength should, if reasoned about correctly, leave belief close to where it started.
• gain_independent / gain_dependent / discounting_gap (dependence discounting) - gain_independent is the increase in confidence from 3 independent confirming sources over 1 source; gain_dependent is the same for 3 sources that are explicitly copies of one another. discounting_gap = gain_independent - gain_dependent. A large positive gap means the model correctly treats copies as weaker evidence than independent corroboration; a gap near or below zero means it does not.
• Dependence Brier - the squared gain_dependent value (as a probability), since a Bayes-ideal updater should gain exactly zero confidence from redundant copies of one source.
• correct_direction (temporal override) - whether the model correctly rates its "as of the original date" confidence higher than its "right now" confidence, after an explicit, unambiguous correction. distractor_condition (direct / with_distractor) marks whether an irrelevant, content-free sentence was inserted between the correction and the question.
• Temporal Brier - squared deviation from two fixed target values (then = 80, now = 15), chosen to reflect how convincing the designed evidence is meant to be. These are a stated design assumption, not derived ground truth.
• Cat1 / Cat2 / Cat3 / Cat4 (numerical fidelity) - four ways an exact numeric value can be corrupted when restated: Cat1 (control) is an ordinary value (any error is the baseline noise floor); Cat2 (latent pull) is a real but slightly unusual value near a well-known constant (does the restated value drift toward the constant?); Cat3 (awkward decimal) is a value with many decimal places (does the tail get truncated?); Cat4 (obfuscated) is a value replaced with an explicit missing-data placeholder (does the model invent a specific number instead of reporting it as missing?). Cat1-3 can only be measured under sequential, since those categories require the model to have actually been shown the value.
• 95% bootstrap confidence interval (CI) - the range that a metric's value would plausibly fall in if the same-sized sample of items were redrawn at random, estimated by resampling the existing results (with replacement) 2,000 times and taking the middle 95% of the resulting means. An interval that excludes 0, or two conditions whose intervals don't overlap, is evidence the difference is unlikely to be sampling noise at this sample size; an interval that straddles 0, or heavily overlapping intervals, means the point estimate should be treated as suggestive only.
• parsed / exact_format / rescued - bookkeeping on how each response was scored. parsed: whether a usable confidence/value could be extracted at all. exact_format: whether the response used the literally requested format (e.g. CONFIDENCE: 75) rather than a variant the parser was made tolerant of (e.g. Confidence: 75). rescued (numeric only): whether the model failed to state its restated values in its first response and needed one follow-up turn, with its reply pre-filled to force the required format while leaving the value itself up to the model.

Appendix A: Full results heatmap

Every cell is a failure rate or normalized error (higher = worse), except the dependence-gap row, where lower is worse (see the key below). Blank cells under isolated for Cat1-Cat3 are expected, not missing data: that condition only ever shows the model the obfuscated Cat4 row in isolation, so Cat1-3 fidelity can only be measured in sequential, where the model sees the full data table.

Row key (exact values, from results/summary_metrics.json):

Appendix B: Full bootstrap confidence intervals

95% bootstrap CIs (2,000 resamples) on every metric above, from results/bootstrap_ci.json and results/dependence_gains_breakdown.csv. n is the number of independent items/pairs the interval is built from (15 for the three epistemic batteries, 24 for numeric).

Order robustness

Dependence discounting

Dependence discounting - gains broken down separately (follow-up check for finding #4)

Temporal override

Numerical fidelity (Cat1-3 only measurable under sequential; see Appendix A)

Appendix C: Cold-baseline results

The model's stated confidence with zero supporting evidence, by source battery and condition (results/summary_metrics.json -> cold_baseline_mean_confidence_by_condition). A well-behaved, uncontaminated fictional-entity design would land close to 50 (a coin flip) in every cell; the spread below shows this did not fully hold, which adds noise to the order-robustness Brier score specifically, since that score is normed against each entity's own measured cold-baseline value.

Mean shift from adding an irrelevant distractor alone (isolated -> sequential, zero real evidence either way; nonzero = anchoring on noise not content): strict = -2.733, weak = 2.222.

Appendix D: Full parse / format-compliance / rescue rates

From results/summary_metrics.json -> parse_rate_by_battery_condition. parsed = fraction of responses a usable value could be extracted from at all; exact_format = fraction following the literally requested format on the first try; rescued (numeric only) = fraction needing the prefilled follow-up turn described in the Important Note on Methodology above.

Appendix E: Numeric fidelity by domain

The numeric battery spans three domains (8 items each per condition). Breaking Cat1-4 down by domain, rather than only by prompt style and context texture, shows two of the four failure categories are not spread evenly across domains:

• Cat2 (drift toward a canonical value) and Cat3 (decimal truncation) occurred exclusively in the chemistry domain in this run - zero instances in physics or finance. This sharpens finding #5 above: these two failure types are not just rare and condition-specific, they are so far domain-specific as well, on this small a sample.
• Cat4 (imputation of a missing value) occurred in every domain, but at different rates: physics shows the highest imputation rate in three of the four conditions tested, finance the lowest and most stable.

At n=8 per domain-condition cell, these are directional observations, not statistically established differences - a larger numeric sample per domain would be needed to confirm the chemistry-specificity of Cat2/Cat3 or the physics/finance gap in Cat4 imputation rates.

References

The following are cited as the specific existing literatures this diagnosis's four failure modes sit closest to:

• Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022). Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 8086-8098.
  • order sensitivity in in-context learning; the closest existing-literature anchor for the order-robustness battery here.
• Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics, 12, 157-173. (arXiv:2307.03172)
  • documents primacy/recency-shaped position sensitivity in long-context use, directly relevant to the sequential-context conditions and the recency finding reported above.
• Kadavath, S., Conerly, T., Askell, A., et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
  • calibration and self-evaluation of stated confidence in language models; the general calibration/uncertainty literature this study's Brier-scoring approach sits within.
• Graves, A., Wayne, G., Reynolds, M., et al. (2016). Hybrid Computing Using a Neural Network with Dynamic External Memory. Nature, 538(7626), 471-476. (Differentiable Neural Computer)
  • together with the earlier Neural Turing Machine (Graves, Wayne & Danihelka, 2014, arXiv:1410.5401), the external-memory-architecture lineage that this study's proposed persistent-substrate direction (Section 3) sits closest to.
• Kirkpatrick, J., Pascanu, R., Rabinowitz, N., et al. (2017). Overcoming Catastrophic Forgetting in Neural Networks. PNAS, 114(13), 3521-3526. (arXiv:1612.00796)
  • the continual-learning (Elastic Weight Consolidation) baseline family relevant to any persistent-belief-state proposal that must avoid overwriting prior knowledge when updated.
• Alchourron, C. E., Gardenfors, P., & Makinson, D. (1985). On the Logic of Theory Change: Partial Meet Contraction and Revision Functions. Journal of Symbolic Logic, 50(2), 510-530.
  • the formal (AGM) belief-revision literature that the temporal-override battery's notion of "correctly overturning a prior belief" is an informal, empirical instance of.

Repository contents

Reproducibility: MODEL_ID = Qwen/Qwen2.5-1.5B-Instruct, SEED = 0, N_PER_CONDITION = 15, N_NUMERIC_VARIANTS = 8, TEMPORAL_TARGET_THEN = 80, TEMPORAL_TARGET_NOW = 15 (a stated design assumption, not derived ground truth).

Limitations

• Single model, single seed. These results characterize one 1.5B instruction-tuned model at one point in time. A cross-model comparison is planned but not run here.
• Sample size. 15 items/pairs per epistemic condition and 24 per numeric condition are enough to detect the headline effects above as distinguishable from noise (per the bootstrap intervals), but several secondary observations (Cat2, Cat3, the strict+sequential dependence result) remain underpowered.
• Hand-authored packets. All evidence packets use a fixed, hand-chosen evidence-strength design and fictional-entity naming scheme; results may not transfer to naturally occurring text.
• This is a diagnosis of monolithic, in-context composition only. No decomposed architecture is built or compared against in this submission - see Section 3.