app/src/content/article.mdx

---
seoThumbImage: 'https://seongland.com/article/optimismbench/og.png'
title: 'OptimismBench'
subtitle: >-
  Forecasting Bias and the Alignment Effect in Language Model Judgment
description: 'Ask a model for the probability something succeeds, then for the probability it fails. The two numbers rarely sum to 100, and the gap leans one way. An interactive look at directional bias in LLM probability judgment.'
authors:
  - name: Seonglae Cho
    url: 'https://openreview.net/profile?id=~Seonglae_Cho2'
    affiliations:
      - 1
      - 2
  - name: Adriano Koshiyama
    affiliations:
      - 1
      - 2
affiliations:
  - name: Holistic AI
    url: 'https://holisticai.com'
  - name: University College London
    url: 'https://ucl.ac.uk'
published: 'Jun. 30, 2026'
paperUrl: 'https://openreview.net/forum?id=sgB6Jwgjod'
citationText: >-
  Cho, S., & Koshiyama, A. (2026). OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment. Forecasting as a New Frontier of Intelligence (ICML 2026 Workshop).
bibtex: >-
  @inproceedings{cho2026optimismbench,
    title={OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgment},
    author={Seonglae Cho and Adriano Koshiyama},
    booktitle={Forecasting as a New Frontier of Intelligence},
    year={2026},
    url={https://openreview.net/forum?id=sgB6Jwgjod}
  }
licence: >
  Text and interactive visualizations are licensed under <a
  href="https://creativecommons.org/licenses/by/4.0/" target="_blank"
  rel="noopener noreferrer">CC-BY 4.0</a>.
doi: ''
tags:
  - large language models
  - forecasting
  - optimism bias
  - alignment
tableOfContentsAutoCollapse: true
pdfProOnly: false
---

import HtmlEmbed from '../components/HtmlEmbed.astro'
import Note from '../components/Note.astro'
import Glossary from '../components/Glossary.astro'
import Accordion from '../components/Accordion.astro'
import Quote from '../components/Quote.astro'
import Sidenote from '../components/Sidenote.astro'
import Wide from '../components/Wide.astro'

## TL;DR

Ask a [language model](https://texonom.com/language-model-577d47779c5f4da98e3fb0df753b9e81) for the probability a startup succeeds. Then, separately, ask for the probability it fails. A coherent judge returns two numbers that sum to 100. They almost never do, and the gap leans one way.

<Note variant="success" title="The one-line claim">
  Across 16 models, 14 are optimistic and only Anthropic's two frontier models are pessimistic. The tilt is not noise: a
  controlled base-versus-chat probe shows [alignment](https://texonom.com/ai-alignment-f676f1a29ffd45e19b3d170afa4f2244)
  training installs it, and across languages model identity dominates over language.
</Note>

## The problem: direction hides from calibration

The usual lens for "does this model know its probabilities" is [calibration](https://texonom.com/ai-confidence-2d3c3c96247d803ba95ce8fb07ae7f2c), and it is blind here. <Glossary term="ECE" definition="Expected Calibration Error: the average gap between a model's stated confidence and its accuracy, aggregated as an unsigned error.">ECE</Glossary> aggregates _unsigned_ error [@guo2017calibration], so a model with ECE = 5 could be uniformly optimistic, uniformly pessimistic, or neither. The sign, the very thing that tells you whether a forecasting agent will over- or under-promise, is averaged away.

The second problem is ground truth. In naturalistic forecasting ("will this person stick to their new exercise routine?") there is no label to score against. So OptimismBench scores the model against _itself_: [@zhu2024incoherent] showed language models violate basic probability axioms, and we keep the sign of one such violation as the signal.

## Inverted pairs

For each scenario we elicit two probabilities, the positive framing and its complement, and define the <Glossary term="Skew" definition="Skew = P(positive) − (100 − P(negative)). A coherent model scores zero; positive Skew is optimism, negative is pessimism.">Skew</Glossary> as the gap between them. Drag the two estimates and watch the unaccounted-for points appear.

<HtmlEmbed
  src="inverted-pair.html"
  title="Skew calculator"
  desc="Figure 1. The inverted-pair method. Skew is the probability mass that goes missing when the two framings disagree."
/>

The construction needs no ground truth and it cancels acquiescence: a model that simply agrees with both framings lands at zero. The paired-complement trick is borrowed from human psychophysics [@kahneman1979prospect]; the contribution here is keeping the _sign_ and reading it as directional valence.

### Three real cases

The metric is abstract; the failure is not. Here are three scenarios from the benchmark exactly as posed, answered by an optimistic model (GPT-5.4) and a pessimistic one (Sonnet 4.6). The numbers are the models' actual elicited probabilities. Switch domains and watch the same text pull the two models in opposite directions.

<HtmlEmbed
  src="case-study.html"
  title="Case studies"
  desc="Real Track B elicitations: GPT-5.4 versus Sonnet 4.6 on three identical scenarios. Same question, opposite tilt."
/>

On the home-exercise scenario the two framings should sum to 100. GPT-5.4 answers 58 and 62, so it leaves twenty points double-counted toward _sticking with it_. Sonnet answers 31 and 55, fourteen points double-counted toward _quitting_. Neither model is wrong about any single number you could check; the bias only appears when you ask the same question both ways and keep the sign.

## Directional bias is pervasive

Every one of the 16 headline models shows nonzero Skew. The pattern spans US commercial APIs, Chinese labs, and European releases. Only Anthropic's Opus and Sonnet land below zero. The full table, with each row tinted by the magnitude of its bias, is the headline result.

<HtmlEmbed
  src="skew-table.html"
  title="Skew table"
  desc="Table 1. Track B Skew, per-item σ, and direction for all 16 models, rows tinted by |Skew|."
/>

At 95% confidence, the two pessimistic rows do not overlap zero, and neither do the optimists.

<Wide>
  <HtmlEmbed
    src="skew-forest.html"
    title="Skew forest plot"
    desc="Figure 2. Track B Skew with 95% CIs across 16 models. Toggle between ranking by Skew and grouping by provider."
  />
</Wide>

The smallest, cheapest models tend to be the most optimistic, and they are exactly the ones deployed at scale for cost. Human optimism bias is a well-studied trait [@weinstein1980unrealistic; @sharot2011optimism]; these models inherit its shape.

## Two ways to be biased

Two models with the same Skew can be biased through different mechanisms. Split it into two components, the good-side push and the bad-side push, each measuring how far one estimate sits above 50, and the mechanism shows.

<HtmlEmbed
  src="valence-quadrant.html"
  title="Valence decomposition"
  desc="Figure 3. Good-side push versus bad-side push for six models. Arrows trace each provider's small-to-large shift."
/>

Optimistic models like GPT-5.4-mini inflate _both_ sides. Sonnet's pessimism is one-sided: it underestimates good outcomes while keeping bad-outcome estimates near accurate. The bias shows up as suppressed opportunity rather than inflated risk.

## Alignment is a lever

The within-provider gradient (smaller is more optimistic) is suggestive but confounded. To isolate post-training, OptimismBench runs controlled base-versus-chat pairs: same architecture, same pre-training, only the alignment step changes.

<HtmlEmbed
  src="alignment-slope.html"
  title="Base vs chat"
  desc="Figure 4. Base versus chat Skew across architectures. Qwen all shift down, Llama all shift up, Gemma flips."
/>

The shift is real and recipe-specific: every [Qwen](https://texonom.com/qwen-98e9c8006e764682ab87af31c84d6bbf) pair moves toward pessimism, every [Llama](https://texonom.com/f2b6721202d44d469add84d8a366809c) pair toward optimism over a pessimistic base, and [Gemma](https://texonom.com/8aa07c8eba2c4d4ba61213b9a4f711c5) flips from strongly pessimistic to strongly optimistic. Alignment sets the direction; it is not a perturbation on a pre-existing fixed bias.

<Accordion title="What this does not show">
  The four headline pairs span two families and one recipe each, so this is causal *within* those families, not a universal
  law. We treat the broader within-provider gradient as an empirical regularity to explain, not a mechanism we have pinned
  down.
</Accordion>

## Model, not language

Does the bias come from a language's training corpus or from the model? Eleven models across six native-prompt languages answer cleanly.

<HtmlEmbed
  src="crosslingual.html"
  title="Cross-lingual heatmap"
  desc="Figure 5. Cross-lingual Skew across 11 models and 6 languages. Read by row and by column: rows vary, columns barely move."
/>

Rows (models) differ wildly; columns (languages) barely move. The average spread between models within a language is 3.3 times the spread between languages within a model. Model identity dominates, and the language-level positivity gradient seen in human corpora does not transfer.

Widen the panel to every model with full six-language coverage, which adds three models beyond the 16-model headline set.<Sidenote>The cross-lingual panel adds Qwen3-32B, Nemotron-3-super, and Gemma-4-31B-IT, which are not in the headline table. So the count of pessimists here is not the headline "two": the headline claim is about the 16-model set, where the only two below zero are Anthropic's Opus and Sonnet.</Sidenote> Collapsing each row to two numbers, its mean Skew and how much that Skew wobbles across the six languages, sorts the whole fleet into four corners. The useful reading is the bottom band: a model can be both biased _and_ stable, the worst case for a deployer, because the tilt is large and cannot be averaged away by switching prompt language.

<HtmlEmbed
  src="robustness.html"
  title="Bias-stability plane"
  desc="Figure 6. Mean Skew versus inter-language σ for 17 models. Right of the line is optimist; lower is more language-robust. The strongly pessimistic, language-stable corner belongs to Anthropic."
/>

Almost everything lands on the right (optimist) and low (robust): the bias is large and it does not wash out across languages. The pessimist side is sparse. Anthropic's Sonnet and Opus are the two strong pessimists and also the most language-stable in the whole set (σ = 0.54 and 0.89). The only other model left of zero is Qwen3-32B, barely past neutral. The volatile corner is mostly small or mid-tier models, so instability across languages is the exception.

## Why it matters for forecasting

Language models are being wired into forecasting pipelines, approaching human crowd accuracy on [prediction markets](https://texonom.com/prediction-market-1f9c3c96247d8083a7f1d24d41487e2f) [@halawi2024approaching]. A pipeline built on a +13 Skew model will systematically overestimate positive outcomes by roughly 13 points, and no aggregate calibration score on that model will warn you. Worse, users overrely on confident model outputs [@rathi2025overrely], so the tilt propagates. Skew is cheap to compute, needs no labels, and works as a model-card number: pick the model whose bias profile matches your risk tolerance.

## Explore the data

The benchmark and the full response corpus are released on HuggingFace. Browse it directly below: pick any of the 60 scenarios to see how the models split it, or pick a model to see its bias profile across every scenario. Switch the prompt language to read the native-language runs (English carries all nine models; the other languages carry the six with full multilingual coverage), and toggle individual models on or off. The per-model means are computed from the released response corpus.

<Wide>
  <HtmlEmbed
    src="dataset-explorer.html"
    title="Dataset explorer"
    desc="Mean P(positive)/P(negative) per Track B scenario per model, with directional Skew. Browse by scenario or by model, switch prompt language, and toggle models."
  />
</Wide>

## Limitations

The cross-lingual evidence covers ten languages plus a six-model confirmation; four languages use a mixed-prompt setup, so the language-versus-prompt attribution is approximate for those entries. Skew measures directional self-inconsistency, not deviation from real-world outcomes, so it captures internal coherence rather than calibration against reality. The 60 scenarios are author-constructed and passed one round of internal review; external inter-rater validation is planned for the expanded release.

## Conclusion

Helpfulness and probability direction turn out to be two outputs of the same training signal, and the side-effect is invisible to every standard check. ECE will not see it, a leaderboard score will not see it, and the model will state a 70% with the same fluency whether or not the complement says 15%.

The uncomfortable part is the ranking. The safest-sounding models are not the most neutral: Anthropic's frontier pair is the only one that leans pessimistic, the small and cheap models lean most optimistic, and the gap between two models on the same question can reach 30 points. Alignment did not remove the bias, it chose its direction. Before you trust a model's odds, ask which way its training bent them; that is a one-number, label-free thing you can now measure.