Spaces:
Paused
Paused
article: interactive research piece for OptimismBench
Browse files- Embeds: inverted-pair, skew-table, skew-forest, valence-quadrant (6 paired
models, provider palette, no diagonal), alignment-slope (Qwen/Llama/Gemma),
crosslingual heatmap, bias-stability plane, real-case study.
- valence: B_en_003-corrected good/bad values.
- Frontmatter authors/affiliations/bibtex; bibliography verbatim from paper;
ARR title in subtitle. Remove dark hero.
- app/src/components/Hero.astro +0 -3
- app/src/content/article.mdx +33 -15
- app/src/content/assets/data/alignpair.json +13 -13
- app/src/content/assets/data/cases.json +36 -0
- app/src/content/assets/data/crosslingual.json +3 -3
- app/src/content/assets/data/robustness.json +22 -0
- app/src/content/assets/data/skew_models.json +18 -17
- app/src/content/assets/data/valence.json +9 -9
- app/src/content/bibliography.bib +30 -36
- app/src/content/embeds/alignment-slope.html +3 -3
- app/src/content/embeds/case-study.html +88 -0
- app/src/content/embeds/crosslingual.html +4 -1
- app/src/content/embeds/hero-axis.html +0 -87
- app/src/content/embeds/robustness.html +108 -0
- app/src/content/embeds/skew-forest.html +6 -1
- app/src/content/embeds/valence-quadrant.html +37 -18
app/src/components/Hero.astro
CHANGED
|
@@ -96,9 +96,6 @@ const pdfFilename = `${slugify(pdfBase)}.pdf`;
|
|
| 96 |
<section class="hero">
|
| 97 |
<h1 class="hero-title" set:html={title} />
|
| 98 |
{description && <p class="hero-desc">{description}</p>}
|
| 99 |
-
<div class="hero-banner">
|
| 100 |
-
<HtmlEmbed src="hero-axis.html" frameless />
|
| 101 |
-
</div>
|
| 102 |
</section>
|
| 103 |
|
| 104 |
<header class="meta" aria-label="Article meta information">
|
|
|
|
| 96 |
<section class="hero">
|
| 97 |
<h1 class="hero-title" set:html={title} />
|
| 98 |
{description && <p class="hero-desc">{description}</p>}
|
|
|
|
|
|
|
|
|
|
| 99 |
</section>
|
| 100 |
|
| 101 |
<header class="meta" aria-label="Article meta information">
|
app/src/content/article.mdx
CHANGED
|
@@ -1,24 +1,34 @@
|
|
| 1 |
---
|
| 2 |
title: 'OptimismBench'
|
| 3 |
subtitle: >-
|
| 4 |
-
Measuring Forecasting Bias in Language
|
| 5 |
description: 'Ask a model for the probability something succeeds, then for the probability it fails. The two numbers rarely sum to 100, and the gap leans one way. An interactive look at directional bias in LLM probability judgment.'
|
| 6 |
authors:
|
| 7 |
- name: Seonglae Cho
|
| 8 |
-
url: 'https://
|
| 9 |
affiliations:
|
| 10 |
- 1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
affiliations:
|
|
|
|
|
|
|
| 12 |
- name: University College London
|
| 13 |
url: 'https://ucl.ac.uk'
|
| 14 |
published: 'Jun. 2026'
|
|
|
|
| 15 |
citationText: >-
|
| 16 |
-
|
| 17 |
bibtex: >-
|
| 18 |
-
@misc{
|
| 19 |
-
title={OptimismBench: Measuring Forecasting Bias in Language
|
| 20 |
-
author={
|
| 21 |
-
year={2026}
|
|
|
|
| 22 |
}
|
| 23 |
licence: >
|
| 24 |
Text and interactive visualizations are licensed under <a
|
|
@@ -64,6 +74,14 @@ For each scenario we elicit two probabilities, the positive framing and its comp
|
|
| 64 |
|
| 65 |
The construction needs no ground truth and it cancels acquiescence: a model that simply agrees with both framings lands at zero. The paired-complement trick is borrowed from human psychophysics [@kahneman1979prospect]; the contribution here is keeping the *sign* and reading it as directional valence.
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
## Directional bias is pervasive
|
| 68 |
|
| 69 |
Every one of the 16 headline models shows significant Skew. The pattern spans US commercial APIs, Chinese labs, and European releases. Only Anthropic's Opus and Sonnet land below zero. The full table, with each row tinted by the magnitude of its bias, is the headline result.
|
|
@@ -104,26 +122,26 @@ The four headline pairs span two families and one recipe each, so this is causal
|
|
| 104 |
|
| 105 |
Does the bias come from a language's training corpus or from the model? Eleven models across six native-prompt languages answer cleanly.
|
| 106 |
|
| 107 |
-
<Wide>
|
| 108 |
<HtmlEmbed src="crosslingual.html" title="Cross-lingual heatmap" desc="Figure 5. Cross-lingual Skew across 11 models and 6 languages. Read by row and by column: rows vary, columns barely move." />
|
| 109 |
-
</Wide>
|
| 110 |
|
| 111 |
Rows (models) differ wildly; columns (languages) barely move. The average spread between models within a language is 3.3 times the spread between languages within a model. Model identity dominates, and the language-level positivity gradient seen in human corpora does not transfer.
|
| 112 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 113 |
## Why it matters for forecasting
|
| 114 |
|
| 115 |
Language models are being wired into forecasting pipelines, approaching human crowd accuracy on prediction markets [@halawi2024approaching]. A pipeline built on a +13 Skew model will systematically overestimate positive outcomes by roughly 13 points, and no aggregate calibration score on that model will warn you. Worse, users overrely on confident model outputs [@rathi2025overrely], so the tilt propagates. Skew is cheap to compute, needs no labels, and works as a model-card number: pick the model whose bias profile matches your risk tolerance.
|
| 116 |
|
| 117 |
## Limitations
|
| 118 |
|
| 119 |
-
<Accordion title="The honest boundary of these claims">
|
| 120 |
The cross-lingual evidence is a single ten-language probe plus a six-model confirmation; four of the ten languages use a hybrid English-system-prompt setup that confounds language with prompt mismatch. Skew measures directional self-inconsistency, not deviation from real-world outcomes, so it is internal coherence rather than calibration against reality. The 60 scenarios are author-constructed and internally reviewed without an external inter-rater check, so the benchmark is a pilot, not a saturated instrument.
|
| 121 |
-
</Accordion>
|
| 122 |
|
| 123 |
## Conclusion
|
| 124 |
|
| 125 |
-
Helpfulness and probability direction turn out to be two outputs of the same training signal.
|
| 126 |
|
| 127 |
-
|
| 128 |
-
OptimismBench makes which direction was installed measurable.
|
| 129 |
-
</Quote>
|
|
|
|
| 1 |
---
|
| 2 |
title: 'OptimismBench'
|
| 3 |
subtitle: >-
|
| 4 |
+
Measuring Forecasting Bias and Probing the Alignment Effect in Language Models
|
| 5 |
description: 'Ask a model for the probability something succeeds, then for the probability it fails. The two numbers rarely sum to 100, and the gap leans one way. An interactive look at directional bias in LLM probability judgment.'
|
| 6 |
authors:
|
| 7 |
- name: Seonglae Cho
|
| 8 |
+
url: 'https://openreview.net/profile?id=~Seonglae_Cho2'
|
| 9 |
affiliations:
|
| 10 |
- 1
|
| 11 |
+
- 2
|
| 12 |
+
- name: Adriano Koshiyama
|
| 13 |
+
url: 'https://holisticai.com'
|
| 14 |
+
affiliations:
|
| 15 |
+
- 1
|
| 16 |
+
- 2
|
| 17 |
affiliations:
|
| 18 |
+
- name: Holistic AI
|
| 19 |
+
url: 'https://holisticai.com'
|
| 20 |
- name: University College London
|
| 21 |
url: 'https://ucl.ac.uk'
|
| 22 |
published: 'Jun. 2026'
|
| 23 |
+
paperUrl: 'https://openreview.net/forum?id=kBT1s66UXy'
|
| 24 |
citationText: >-
|
| 25 |
+
Anonymous (2026). OptimismBench: Measuring Forecasting Bias and Probing the Alignment Effect in Language Models. OpenReview.
|
| 26 |
bibtex: >-
|
| 27 |
+
@misc{anonymous2026optimismbench,
|
| 28 |
+
title={OptimismBench: Measuring Forecasting Bias and Probing the Alignment Effect in Language Models},
|
| 29 |
+
author={Anonymous},
|
| 30 |
+
year={2026},
|
| 31 |
+
url={https://openreview.net/forum?id=kBT1s66UXy}
|
| 32 |
}
|
| 33 |
licence: >
|
| 34 |
Text and interactive visualizations are licensed under <a
|
|
|
|
| 74 |
|
| 75 |
The construction needs no ground truth and it cancels acquiescence: a model that simply agrees with both framings lands at zero. The paired-complement trick is borrowed from human psychophysics [@kahneman1979prospect]; the contribution here is keeping the *sign* and reading it as directional valence.
|
| 76 |
|
| 77 |
+
### Three real cases
|
| 78 |
+
|
| 79 |
+
The metric is abstract; the failure is not. Here are three scenarios from the benchmark exactly as posed, answered by an optimistic model (GPT-5.4) and a pessimistic one (Sonnet 4.6). The numbers are the models' actual elicited probabilities. Switch domains and watch the same text pull the two models in opposite directions.
|
| 80 |
+
|
| 81 |
+
<HtmlEmbed src="case-study.html" title="Case studies" desc="Real Track B elicitations: GPT-5.4 versus Sonnet 4.6 on three identical scenarios. Same question, opposite tilt." />
|
| 82 |
+
|
| 83 |
+
On the home-exercise scenario the two framings should sum to 100. GPT-5.4 answers 58 and 62 — it leaves twenty points double-counted toward *sticking with it*. Sonnet answers 31 and 55, fourteen points double-counted toward *quitting*. Neither model is wrong about any single number you could check; the bias only appears when you ask the same question both ways and keep the sign.
|
| 84 |
+
|
| 85 |
## Directional bias is pervasive
|
| 86 |
|
| 87 |
Every one of the 16 headline models shows significant Skew. The pattern spans US commercial APIs, Chinese labs, and European releases. Only Anthropic's Opus and Sonnet land below zero. The full table, with each row tinted by the magnitude of its bias, is the headline result.
|
|
|
|
| 122 |
|
| 123 |
Does the bias come from a language's training corpus or from the model? Eleven models across six native-prompt languages answer cleanly.
|
| 124 |
|
|
|
|
| 125 |
<HtmlEmbed src="crosslingual.html" title="Cross-lingual heatmap" desc="Figure 5. Cross-lingual Skew across 11 models and 6 languages. Read by row and by column: rows vary, columns barely move." />
|
|
|
|
| 126 |
|
| 127 |
Rows (models) differ wildly; columns (languages) barely move. The average spread between models within a language is 3.3 times the spread between languages within a model. Model identity dominates, and the language-level positivity gradient seen in human corpora does not transfer.
|
| 128 |
|
| 129 |
+
Widen the panel to every model with full six-language coverage — which adds three models beyond the 16-model headline set<Sidenote>The cross-lingual panel adds Qwen3-32B, Nemotron-3-super, and Gemma-4-31B-IT, which are not in the headline table. So the count of pessimists here is not the headline "two": the headline claim is about the 16-model set, where the only two below zero are Anthropic's Opus and Sonnet.</Sidenote> — and collapse each row to two numbers: its mean Skew and how much that Skew wobbles across the six languages. The whole fleet sorts into four corners, and the useful reading is the bottom band: a model can be both biased *and* stable, which is the worst case for a deployer, because the tilt is large and you cannot average it away by switching prompt language.
|
| 130 |
+
|
| 131 |
+
<HtmlEmbed src="robustness.html" title="Bias-stability plane" desc="Figure 6. Mean Skew versus inter-language σ for 17 models. Right of the line is optimist; lower is more language-robust. The strongly pessimistic, language-stable corner belongs to Anthropic." />
|
| 132 |
+
|
| 133 |
+
Almost everything lands on the right (optimist) and low (robust): the bias is large and it does not wash out across languages. The pessimist side is sparse — Anthropic's Sonnet and Opus are the two strong pessimists, and they are also the most language-stable models in the whole set (σ = 0.54 and 0.89); the only other model left of zero is the small Qwen3-32B, drifting barely past neutral. The volatile corner is mostly small or mid-tier models; instability across languages is the exception, not the rule.
|
| 134 |
+
|
| 135 |
## Why it matters for forecasting
|
| 136 |
|
| 137 |
Language models are being wired into forecasting pipelines, approaching human crowd accuracy on prediction markets [@halawi2024approaching]. A pipeline built on a +13 Skew model will systematically overestimate positive outcomes by roughly 13 points, and no aggregate calibration score on that model will warn you. Worse, users overrely on confident model outputs [@rathi2025overrely], so the tilt propagates. Skew is cheap to compute, needs no labels, and works as a model-card number: pick the model whose bias profile matches your risk tolerance.
|
| 138 |
|
| 139 |
## Limitations
|
| 140 |
|
|
|
|
| 141 |
The cross-lingual evidence is a single ten-language probe plus a six-model confirmation; four of the ten languages use a hybrid English-system-prompt setup that confounds language with prompt mismatch. Skew measures directional self-inconsistency, not deviation from real-world outcomes, so it is internal coherence rather than calibration against reality. The 60 scenarios are author-constructed and internally reviewed without an external inter-rater check, so the benchmark is a pilot, not a saturated instrument.
|
|
|
|
| 142 |
|
| 143 |
## Conclusion
|
| 144 |
|
| 145 |
+
Helpfulness and probability direction turn out to be two outputs of the same training signal, and the side-effect is invisible to every standard check. ECE will not see it, a leaderboard score will not see it, and the model will state a 70% with the same fluency whether or not the complement says 15%.
|
| 146 |
|
| 147 |
+
The uncomfortable part is the ranking. The safest-sounding models are not the most neutral: Anthropic's frontier pair is the only one that leans pessimistic, the small and cheap models lean most optimistic, and the gap between two models on the same question can reach 30 points. Alignment did not remove the bias, it chose its direction. Before you trust a model's odds, the useful question is not "is it calibrated" but "which way did its training bend it" — and that is a one-number, label-free thing you can now measure.
|
|
|
|
|
|
app/src/content/assets/data/alignpair.json
CHANGED
|
@@ -1,16 +1,16 @@
|
|
| 1 |
{
|
| 2 |
-
"note":
|
| 3 |
-
"pairs":
|
| 4 |
-
{"arch":
|
| 5 |
-
{"arch":
|
| 6 |
-
{"arch":
|
| 7 |
-
{"arch":
|
| 8 |
-
{"arch":
|
| 9 |
-
{"arch":
|
| 10 |
-
{"arch":
|
| 11 |
-
{"arch":
|
| 12 |
-
{"arch":
|
| 13 |
-
{"arch":
|
| 14 |
-
{"arch":
|
| 15 |
]
|
| 16 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"note":"Controlled base-vs-chat probe. delta=chat-base.",
|
| 3 |
+
"pairs":[
|
| 4 |
+
{"arch":"Qwen2.5-7B","family":"Qwen","base":14.6,"chat":9.0,"delta":-5.6},
|
| 5 |
+
{"arch":"Qwen3-1.7B","family":"Qwen","base":-12.4,"chat":-21.4,"delta":-9.1},
|
| 6 |
+
{"arch":"Qwen3-4B","family":"Qwen","base":0.4,"chat":-15.5,"delta":-15.9},
|
| 7 |
+
{"arch":"Qwen3-8B","family":"Qwen","base":16.1,"chat":2.1,"delta":-14.0},
|
| 8 |
+
{"arch":"Qwen3-14B","family":"Qwen","base":16.1,"chat":-0.1,"delta":-16.2},
|
| 9 |
+
{"arch":"Llama-3.2-1B","family":"Llama","base":-33.7,"chat":7.3,"delta":40.9},
|
| 10 |
+
{"arch":"Llama-3.2-3B","family":"Llama","base":-6.4,"chat":18.7,"delta":25.1},
|
| 11 |
+
{"arch":"Llama-3.1-8B","family":"Llama","base":1.9,"chat":21.9,"delta":20.0},
|
| 12 |
+
{"arch":"Llama-3.1-70B","family":"Llama","base":-3.1,"chat":12.2,"delta":15.2},
|
| 13 |
+
{"arch":"Gemma-2-2b","family":"Gemma","base":-17.5,"chat":36.0,"delta":53.5},
|
| 14 |
+
{"arch":"Mistral-Small-24B","family":"Mistral","base":8.0,"chat":5.2,"delta":-2.8}
|
| 15 |
]
|
| 16 |
}
|
app/src/content/assets/data/cases.json
ADDED
|
@@ -0,0 +1,36 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"note": "Real Track B elicitations. p_good = P(positive framing), p_bad = P(negative framing). Skew = p_good - (100 - p_bad). GPT-5.4 is an optimist (mean Skew +10.0), Sonnet 4.6 a pessimist (mean -7.7).",
|
| 3 |
+
"models": [
|
| 4 |
+
{ "name": "GPT-5.4", "provider": "OpenAI", "dir": "optimist" },
|
| 5 |
+
{ "name": "Sonnet 4.6", "provider": "Anthropic", "dir": "pessimist" }
|
| 6 |
+
],
|
| 7 |
+
"cases": [
|
| 8 |
+
{
|
| 9 |
+
"id": "B_en_031",
|
| 10 |
+
"domain": "Health",
|
| 11 |
+
"scenario": "A person has started a simple home exercise routine after work three times a week. They often feel tired in the evening, but they also say they want to build a more regular habit.",
|
| 12 |
+
"pos": "What is the probability that this person keeps the exercise routine for at least the next two months?",
|
| 13 |
+
"neg": "What is the probability that this person stops the routine within the next two months?",
|
| 14 |
+
"GPT-5.4": { "good": 58.0, "bad": 62.0 },
|
| 15 |
+
"Sonnet 4.6": { "good": 31.0, "bad": 55.0 }
|
| 16 |
+
},
|
| 17 |
+
{
|
| 18 |
+
"id": "B_en_011",
|
| 19 |
+
"domain": "Business",
|
| 20 |
+
"scenario": "A small company has developed a new household cleaning product and is considering a regional launch. Early customer interviews were mixed, and the company has budget for only one main marketing campaign.",
|
| 21 |
+
"pos": "What is the probability that the product hits its first-year sales target?",
|
| 22 |
+
"neg": "What is the probability that the product does not hit its first-year sales target?",
|
| 23 |
+
"GPT-5.4": { "good": 39.2, "bad": 67.4 },
|
| 24 |
+
"Sonnet 4.6": { "good": 35.0, "bad": 60.0 }
|
| 25 |
+
},
|
| 26 |
+
{
|
| 27 |
+
"id": "B_en_051",
|
| 28 |
+
"domain": "Project",
|
| 29 |
+
"scenario": "A project manager is coordinating several departments to deliver a client report. Some sections are done; others depend on data not yet finalized, and the team has limited time for review.",
|
| 30 |
+
"pos": "What is the probability that the team submits the full report on time?",
|
| 31 |
+
"neg": "What is the probability that the team does not submit on time?",
|
| 32 |
+
"GPT-5.4": { "good": 43.2, "bad": 67.8 },
|
| 33 |
+
"Sonnet 4.6": { "good": 40.4, "bad": 56.0 }
|
| 34 |
+
}
|
| 35 |
+
]
|
| 36 |
+
}
|
app/src/content/assets/data/crosslingual.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
{
|
| 2 |
-
"note":
|
| 3 |
-
"languages":
|
| 4 |
-
"models":
|
| 5 |
{"name":"GLM-4.7-flash","skew":[16.0,14.1,12.3,17.2,17.4,16.1]},
|
| 6 |
{"name":"Mistral Small","skew":[9.7,15.6,20.0,15.2,14.9,16.9]},
|
| 7 |
{"name":"Qwen3-Next-80B","skew":[9.8,8.4,11.5,15.7,15.3,15.5]},
|
|
|
|
| 1 |
{
|
| 2 |
+
"note":"Cross-lingual Skew, 11 models x 6 native-prompt languages (OptimismBench Table 7).",
|
| 3 |
+
"languages":["EN","KO","ZH","ES","AR","RU"],
|
| 4 |
+
"models":[
|
| 5 |
{"name":"GLM-4.7-flash","skew":[16.0,14.1,12.3,17.2,17.4,16.1]},
|
| 6 |
{"name":"Mistral Small","skew":[9.7,15.6,20.0,15.2,14.9,16.9]},
|
| 7 |
{"name":"Qwen3-Next-80B","skew":[9.8,8.4,11.5,15.7,15.3,15.5]},
|
app/src/content/assets/data/robustness.json
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"note": "Mean Skew across 6 native-prompt languages (x) vs inter-language sigma (y), per model. Paper fig_alignment_stability.",
|
| 3 |
+
"models": [
|
| 4 |
+
{ "name": "GLM-4.7-flash", "skew": 15.97, "sigma": 2.34, "provider": "Zhipu" },
|
| 5 |
+
{ "name": "Mistral Small", "skew": 15.42, "sigma": 3.41, "provider": "Mistral" },
|
| 6 |
+
{ "name": "DeepSeek-V3.2", "skew": 13.78, "sigma": 1.69, "provider": "DeepSeek" },
|
| 7 |
+
{ "name": "Qwen3-Next-80B", "skew": 12.92, "sigma": 3.16, "provider": "Alibaba" },
|
| 8 |
+
{ "name": "GPT-5.4-mini", "skew": 12.35, "sigma": 2.10, "provider": "OpenAI" },
|
| 9 |
+
{ "name": "Llama-3.3-70B", "skew": 12.00, "sigma": 2.63, "provider": "Meta" },
|
| 10 |
+
{ "name": "Qwen3-235B", "skew": 11.10, "sigma": 1.31, "provider": "Alibaba" },
|
| 11 |
+
{ "name": "Mistral Large", "skew": 10.82, "sigma": 1.22, "provider": "Mistral" },
|
| 12 |
+
{ "name": "GPT-5.4", "skew": 10.49, "sigma": 0.95, "provider": "OpenAI" },
|
| 13 |
+
{ "name": "Gemini Flash 3", "skew": 6.48, "sigma": 0.92, "provider": "Google" },
|
| 14 |
+
{ "name": "Haiku 4.5", "skew": 6.30, "sigma": 1.16, "provider": "Anthropic" },
|
| 15 |
+
{ "name": "GLM-4.5-Air", "skew": 6.12, "sigma": 2.62, "provider": "Zhipu" },
|
| 16 |
+
{ "name": "Nemotron-3-super","skew": 5.07, "sigma": 1.85, "provider": "NVIDIA" },
|
| 17 |
+
{ "name": "Gemma-4-31B-IT", "skew": 1.73, "sigma": 0.75, "provider": "Google" },
|
| 18 |
+
{ "name": "Qwen3-32B", "skew": -1.65, "sigma": 1.27, "provider": "Alibaba" },
|
| 19 |
+
{ "name": "Opus 4.6", "skew": -5.38, "sigma": 0.89, "provider": "Anthropic" },
|
| 20 |
+
{ "name": "Sonnet 4.6", "skew": -8.33, "sigma": 0.54, "provider": "Anthropic" }
|
| 21 |
+
]
|
| 22 |
+
}
|
app/src/content/assets/data/skew_models.json
CHANGED
|
@@ -1,21 +1,22 @@
|
|
| 1 |
{
|
| 2 |
-
"note": "Track B Skew
|
|
|
|
| 3 |
"models": [
|
| 4 |
-
{"name":
|
| 5 |
-
{"name":
|
| 6 |
-
{"name":
|
| 7 |
-
{"name":
|
| 8 |
-
{"name":
|
| 9 |
-
{"name":
|
| 10 |
-
{"name":
|
| 11 |
-
{"name":
|
| 12 |
-
{"name":
|
| 13 |
-
{"name":
|
| 14 |
-
{"name":
|
| 15 |
-
{"name":
|
| 16 |
-
{"name":
|
| 17 |
-
{"name":
|
| 18 |
-
{"name":
|
| 19 |
-
{"name":
|
| 20 |
]
|
| 21 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"note": "Track B Skew across 16 headline models (OptimismBench Table 1). 60 inverted pairs each.",
|
| 3 |
+
"n": 60,
|
| 4 |
"models": [
|
| 5 |
+
{"name":"GLM-4.7-flash","provider":"Zhipu","size":"S","skew":16.6,"sigma":11.9,"dir":"Opt"},
|
| 6 |
+
{"name":"Llama 3.3-70B","provider":"Meta","size":"L","skew":13.1,"sigma":10.4,"dir":"Opt"},
|
| 7 |
+
{"name":"GPT-5.4-mini","provider":"OpenAI","size":"S","skew":13.1,"sigma":8.8,"dir":"Opt"},
|
| 8 |
+
{"name":"Mistral Large","provider":"Mistral","size":"L","skew":12.2,"sigma":10.6,"dir":"Opt"},
|
| 9 |
+
{"name":"Qwen3-235B","provider":"Alibaba","size":"L","skew":11.2,"sigma":8.8,"dir":"Opt"},
|
| 10 |
+
{"name":"DeepSeek-V3.2","provider":"DeepSeek","size":"L","skew":10.3,"sigma":7.8,"dir":"Opt"},
|
| 11 |
+
{"name":"GPT-5.4","provider":"OpenAI","size":"L","skew":10.0,"sigma":7.0,"dir":"Opt"},
|
| 12 |
+
{"name":"Qwen3-Next-80B","provider":"Alibaba","size":"L","skew":9.6,"sigma":10.0,"dir":"Opt"},
|
| 13 |
+
{"name":"Mistral Small","provider":"Mistral","size":"S","skew":9.6,"sigma":11.8,"dir":"Opt"},
|
| 14 |
+
{"name":"GPT-OSS-120B","provider":"OpenAI","size":"L","skew":6.3,"sigma":7.6,"dir":"Opt"},
|
| 15 |
+
{"name":"Haiku 4.5","provider":"Anthropic","size":"S","skew":6.1,"sigma":11.7,"dir":"Opt"},
|
| 16 |
+
{"name":"Flash 3","provider":"Google","size":"S","skew":5.1,"sigma":7.2,"dir":"Opt"},
|
| 17 |
+
{"name":"GLM-4.5-Air","provider":"Zhipu","size":"S","skew":5.0,"sigma":8.0,"dir":"Opt"},
|
| 18 |
+
{"name":"Pro 3.1","provider":"Google","size":"L","skew":4.2,"sigma":8.1,"dir":"Opt"},
|
| 19 |
+
{"name":"Opus 4.6","provider":"Anthropic","size":"L","skew":-5.1,"sigma":6.4,"dir":"Pes"},
|
| 20 |
+
{"name":"Sonnet 4.6","provider":"Anthropic","size":"L","skew":-7.7,"sigma":6.8,"dir":"Pes"}
|
| 21 |
]
|
| 22 |
}
|
app/src/content/assets/data/valence.json
CHANGED
|
@@ -1,12 +1,12 @@
|
|
| 1 |
{
|
| 2 |
-
"note":
|
| 3 |
-
"models":
|
| 4 |
-
{"name":
|
| 5 |
-
{"name":
|
| 6 |
-
{"name":
|
| 7 |
-
{"name":
|
| 8 |
-
{"name":
|
| 9 |
-
{"name":
|
| 10 |
],
|
| 11 |
-
"pairs":
|
| 12 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"note":"Valence decomposition (OptimismBench Fig 2). good=P(good)-50, bad=P(bad)-50. B_en_003 valence-label corrected.",
|
| 3 |
+
"models":[
|
| 4 |
+
{"name":"GPT-5.4-mini","provider":"OpenAI","good":7.3,"bad":5.8},
|
| 5 |
+
{"name":"GPT-5.4","provider":"OpenAI","good":4.2,"bad":5.5},
|
| 6 |
+
{"name":"Haiku 4.5","provider":"Anthropic","good":-0.2,"bad":6.4},
|
| 7 |
+
{"name":"Flash 3","provider":"Google","good":3.8,"bad":1.2},
|
| 8 |
+
{"name":"Pro 3.1","provider":"Google","good":-1.9,"bad":5.6},
|
| 9 |
+
{"name":"Sonnet 4.6","provider":"Anthropic","good":-6.6,"bad":-1.1}
|
| 10 |
],
|
| 11 |
+
"pairs":[["GPT-5.4-mini","GPT-5.4"],["Haiku 4.5","Sonnet 4.6"],["Flash 3","Pro 3.1"]]
|
| 12 |
}
|
app/src/content/bibliography.bib
CHANGED
|
@@ -1,27 +1,19 @@
|
|
| 1 |
-
@misc{cho2026optimismbench,
|
| 2 |
-
title={OptimismBench: Measuring Forecasting Bias in Language Model Judgment},
|
| 3 |
-
author={Cho, Seonglae},
|
| 4 |
-
year={2026}
|
| 5 |
-
}
|
| 6 |
-
|
| 7 |
@misc{guo2017calibration,
|
| 8 |
-
title={On Calibration of Modern Neural Networks},
|
| 9 |
author={Chuan Guo and Geoff Pleiss and Yu Sun and Kilian Q. Weinberger},
|
| 10 |
year={2017},
|
| 11 |
eprint={1706.04599},
|
| 12 |
archivePrefix={arXiv},
|
| 13 |
primaryClass={cs.LG},
|
| 14 |
-
url={https://arxiv.org/abs/1706.04599},
|
| 15 |
}
|
| 16 |
|
| 17 |
-
@
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
primaryClass={cs.CL},
|
| 24 |
-
url={https://arxiv.org/abs/2401.16646},
|
| 25 |
}
|
| 26 |
|
| 27 |
@article{kahneman1979prospect,
|
|
@@ -34,14 +26,12 @@
|
|
| 34 |
year={1979}
|
| 35 |
}
|
| 36 |
|
| 37 |
-
@
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
pages={806--820},
|
| 44 |
-
year={1980}
|
| 45 |
}
|
| 46 |
|
| 47 |
@article{sharot2011optimism,
|
|
@@ -54,18 +44,22 @@
|
|
| 54 |
year={2011}
|
| 55 |
}
|
| 56 |
|
| 57 |
-
@
|
| 58 |
-
title={
|
| 59 |
-
author={
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
|
|
|
|
|
|
| 63 |
}
|
| 64 |
|
| 65 |
-
@
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
|
|
|
|
|
|
|
| 71 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
@misc{guo2017calibration,
|
| 2 |
+
title={On Calibration of Modern Neural Networks},
|
| 3 |
author={Chuan Guo and Geoff Pleiss and Yu Sun and Kilian Q. Weinberger},
|
| 4 |
year={2017},
|
| 5 |
eprint={1706.04599},
|
| 6 |
archivePrefix={arXiv},
|
| 7 |
primaryClass={cs.LG},
|
| 8 |
+
url={https://arxiv.org/abs/1706.04599},
|
| 9 |
}
|
| 10 |
|
| 11 |
+
@inproceedings{halawi2024approaching,
|
| 12 |
+
title={Approaching Human-Level Forecasting with Language Models},
|
| 13 |
+
author={Danny Halawi and Fred Zhang and Chen Yueh-Han and Jacob Steinhardt},
|
| 14 |
+
booktitle={The Thirty-eighth Annual Conference on Neural Information Processing Systems},
|
| 15 |
+
year={2024},
|
| 16 |
+
url={https://openreview.net/forum?id=FlcdW7NPRY}
|
|
|
|
|
|
|
| 17 |
}
|
| 18 |
|
| 19 |
@article{kahneman1979prospect,
|
|
|
|
| 26 |
year={1979}
|
| 27 |
}
|
| 28 |
|
| 29 |
+
@inproceedings{rathi2025overrely,
|
| 30 |
+
title={Humans overrely on overconfident language models, across languages},
|
| 31 |
+
author={Neil Rathi and Dan Jurafsky and Kaitlyn Zhou},
|
| 32 |
+
booktitle={Second Conference on Language Modeling},
|
| 33 |
+
year={2025},
|
| 34 |
+
url={https://openreview.net/forum?id=QsQatTzATT}
|
|
|
|
|
|
|
| 35 |
}
|
| 36 |
|
| 37 |
@article{sharot2011optimism,
|
|
|
|
| 44 |
year={2011}
|
| 45 |
}
|
| 46 |
|
| 47 |
+
@article{weinstein1980unrealistic,
|
| 48 |
+
title={Unrealistic Optimism About Future Life Events},
|
| 49 |
+
author={Weinstein, Neil D},
|
| 50 |
+
journal={Journal of Personality and Social Psychology},
|
| 51 |
+
volume={39},
|
| 52 |
+
number={5},
|
| 53 |
+
pages={806--820},
|
| 54 |
+
year={1980}
|
| 55 |
}
|
| 56 |
|
| 57 |
+
@misc{zhu2024incoherent,
|
| 58 |
+
title={Incoherent Probability Judgments in Large Language Models},
|
| 59 |
+
author={Jian-Qiao Zhu and Thomas L. Griffiths},
|
| 60 |
+
year={2025},
|
| 61 |
+
eprint={2401.16646},
|
| 62 |
+
archivePrefix={arXiv},
|
| 63 |
+
primaryClass={cs.CL},
|
| 64 |
+
url={https://arxiv.org/abs/2401.16646},
|
| 65 |
}
|
app/src/content/embeds/alignment-slope.html
CHANGED
|
@@ -2,8 +2,7 @@
|
|
| 2 |
<div class="as-head">
|
| 3 |
<span class="as-title">Base → Chat Skew shift</span>
|
| 4 |
<span class="as-tog">
|
| 5 |
-
<button class="as-btn" data-f="
|
| 6 |
-
<button class="as-btn" data-f="Qwen" aria-pressed="false">Qwen</button>
|
| 7 |
<button class="as-btn" data-f="Llama" aria-pressed="false">Llama</button>
|
| 8 |
<button class="as-btn" data-f="Gemma" aria-pressed="false">Gemma</button>
|
| 9 |
</span>
|
|
@@ -31,7 +30,7 @@
|
|
| 31 |
var tip = document.createElement("div"); tip.className = "as-tip"; root.appendChild(tip);
|
| 32 |
var FAM = { Qwen: "#7c3aed", Llama: "#2563eb", Gemma: "#0891b2", Mistral: "#ea580c" };
|
| 33 |
function cssVar(n, fb) { var v = getComputedStyle(document.documentElement).getPropertyValue(n); return (v && v.trim()) || fb; }
|
| 34 |
-
var DATA = null, FILT = "
|
| 35 |
function draw() {
|
| 36 |
if (!DATA) { return; }
|
| 37 |
wrap.innerHTML = "";
|
|
@@ -45,6 +44,7 @@
|
|
| 45 |
[-30, -15, 0, 15, 30].forEach(function (t) { svg.append("text").attr("x", m.l).attr("y", y(t) + 3).attr("fill", MUT).attr("font-size", 9.5).text(t > 0 ? "+" + t : t); });
|
| 46 |
svg.append("text").attr("transform", "translate(15," + H / 2 + ")rotate(-90)").attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 11).text("Skew");
|
| 47 |
DATA.pairs.forEach(function (p) {
|
|
|
|
| 48 |
var c = FAM[p.family], on = (FILT === "all" || p.family === FILT);
|
| 49 |
var g = svg.append("g").style("cursor", "pointer").attr("opacity", on ? 1 : 0.12);
|
| 50 |
g.append("line").attr("x1", xl).attr("y1", y(p.base)).attr("x2", xr).attr("y2", y(p.chat)).attr("stroke", c).attr("stroke-width", 1.8).attr("opacity", 0.82);
|
|
|
|
| 2 |
<div class="as-head">
|
| 3 |
<span class="as-title">Base → Chat Skew shift</span>
|
| 4 |
<span class="as-tog">
|
| 5 |
+
<button class="as-btn" data-f="Qwen" aria-pressed="true">Qwen</button>
|
|
|
|
| 6 |
<button class="as-btn" data-f="Llama" aria-pressed="false">Llama</button>
|
| 7 |
<button class="as-btn" data-f="Gemma" aria-pressed="false">Gemma</button>
|
| 8 |
</span>
|
|
|
|
| 30 |
var tip = document.createElement("div"); tip.className = "as-tip"; root.appendChild(tip);
|
| 31 |
var FAM = { Qwen: "#7c3aed", Llama: "#2563eb", Gemma: "#0891b2", Mistral: "#ea580c" };
|
| 32 |
function cssVar(n, fb) { var v = getComputedStyle(document.documentElement).getPropertyValue(n); return (v && v.trim()) || fb; }
|
| 33 |
+
var DATA = null, FILT = "Qwen";
|
| 34 |
function draw() {
|
| 35 |
if (!DATA) { return; }
|
| 36 |
wrap.innerHTML = "";
|
|
|
|
| 44 |
[-30, -15, 0, 15, 30].forEach(function (t) { svg.append("text").attr("x", m.l).attr("y", y(t) + 3).attr("fill", MUT).attr("font-size", 9.5).text(t > 0 ? "+" + t : t); });
|
| 45 |
svg.append("text").attr("transform", "translate(15," + H / 2 + ")rotate(-90)").attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 11).text("Skew");
|
| 46 |
DATA.pairs.forEach(function (p) {
|
| 47 |
+
if (p.family === "Mistral") { return; }
|
| 48 |
var c = FAM[p.family], on = (FILT === "all" || p.family === FILT);
|
| 49 |
var g = svg.append("g").style("cursor", "pointer").attr("opacity", on ? 1 : 0.12);
|
| 50 |
g.append("line").attr("x1", xl).attr("y1", y(p.base)).attr("x2", xr).attr("y2", y(p.chat)).attr("stroke", c).attr("stroke-width", 1.8).attr("opacity", 0.82);
|
app/src/content/embeds/case-study.html
ADDED
|
@@ -0,0 +1,88 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<div id="ob-cs" role="img" aria-label="Three real scenarios elicited from an optimistic model (GPT-5.4) and a pessimistic model (Sonnet 4.6). On the same scenario the two models split in opposite directions: GPT leaves positive probability mass uncovered, Sonnet leaves negative mass uncovered.">
|
| 2 |
+
<div class="cs-head">
|
| 3 |
+
<span class="cs-title">Same scenario, opposite tilt</span>
|
| 4 |
+
<span class="cs-tog"></span>
|
| 5 |
+
</div>
|
| 6 |
+
<div class="cs-scen"></div>
|
| 7 |
+
<div class="cs-grid"></div>
|
| 8 |
+
<div class="cs-cap">Two framings of one question. A coherent judge has them sum to 100; the leftover is Skew. GPT-5.4 overcounts the good side, Sonnet 4.6 the bad side — on identical text.</div>
|
| 9 |
+
</div>
|
| 10 |
+
<style>
|
| 11 |
+
#ob-cs { font:13px/1.45 system-ui,sans-serif; color:var(--text-color,#222); width:100%; position:relative; }
|
| 12 |
+
#ob-cs .cs-head { display:flex; align-items:baseline; gap:10px; margin-bottom:10px; }
|
| 13 |
+
#ob-cs .cs-title { font-weight:650; font-size:15px; }
|
| 14 |
+
#ob-cs .cs-tog { display:inline-flex; gap:6px; margin-left:auto; flex-wrap:wrap; }
|
| 15 |
+
#ob-cs .cs-btn { font:inherit; font-size:12px; cursor:pointer; padding:4px 10px; border-radius:7px; border:1px solid var(--border-color,#ccc); background:transparent; color:var(--muted-color,#666); }
|
| 16 |
+
#ob-cs .cs-btn:hover { color:var(--text-color,#222); }
|
| 17 |
+
#ob-cs .cs-btn[aria-pressed="true"] { background:var(--primary-color,#2d2926); color:#fff; border-color:var(--primary-color,#2d2926); }
|
| 18 |
+
#ob-cs .cs-scen { background:var(--surface-bg,#faf8f6); border:1px solid var(--border-color,#e7e2dc); border-radius:10px; padding:12px 14px; margin-bottom:14px; }
|
| 19 |
+
#ob-cs .cs-scen .cs-dom { font-size:11px; font-weight:700; letter-spacing:.04em; text-transform:uppercase; color:var(--muted-color,#888); }
|
| 20 |
+
#ob-cs .cs-scen .cs-txt { margin-top:5px; font-size:13.5px; line-height:1.5; }
|
| 21 |
+
#ob-cs .cs-scen .cs-qs { margin-top:9px; display:flex; gap:16px; flex-wrap:wrap; font-size:12px; color:var(--muted-color,#666); }
|
| 22 |
+
#ob-cs .cs-scen .cs-qs .cs-qpos b, #ob-cs .cs-scen .cs-qs .cs-qneg b { font-weight:650; }
|
| 23 |
+
#ob-cs .cs-grid { display:grid; grid-template-columns:1fr 1fr; gap:14px; }
|
| 24 |
+
@media (max-width:560px){ #ob-cs .cs-grid { grid-template-columns:1fr; } }
|
| 25 |
+
#ob-cs .cs-card { border:1px solid var(--border-color,#e7e2dc); border-radius:10px; padding:12px 14px 14px; }
|
| 26 |
+
#ob-cs .cs-mhead { display:flex; align-items:center; gap:8px; margin-bottom:11px; }
|
| 27 |
+
#ob-cs .cs-dot { width:9px; height:9px; border-radius:50%; flex:none; }
|
| 28 |
+
#ob-cs .cs-mname { font-weight:650; font-size:13.5px; }
|
| 29 |
+
#ob-cs .cs-mtag { font-size:11px; color:var(--muted-color,#888); margin-left:auto; }
|
| 30 |
+
#ob-cs .cs-bar { margin:8px 0; }
|
| 31 |
+
#ob-cs .cs-blab { display:flex; justify-content:space-between; font-size:11.5px; margin-bottom:3px; }
|
| 32 |
+
#ob-cs .cs-blab span:last-child { font-variant-numeric:tabular-nums; font-weight:600; }
|
| 33 |
+
#ob-cs .cs-track { height:8px; border-radius:5px; background:var(--border-color,#ece7e1); overflow:hidden; }
|
| 34 |
+
#ob-cs .cs-fill { height:100%; border-radius:5px; }
|
| 35 |
+
#ob-cs .cs-foot { margin-top:11px; padding-top:10px; border-top:1px dashed var(--border-color,#e7e2dc); display:flex; align-items:baseline; gap:8px; }
|
| 36 |
+
#ob-cs .cs-sum { font-size:11.5px; color:var(--muted-color,#777); font-variant-numeric:tabular-nums; }
|
| 37 |
+
#ob-cs .cs-badge { margin-left:auto; font-size:12px; font-weight:700; padding:2px 9px; border-radius:999px; color:#fff; font-variant-numeric:tabular-nums; }
|
| 38 |
+
#ob-cs .cs-cap { margin-top:13px; font-size:12px; color:var(--muted-color,#666); }
|
| 39 |
+
</style>
|
| 40 |
+
<script>
|
| 41 |
+
(function () {
|
| 42 |
+
var root = document.getElementById("ob-cs");
|
| 43 |
+
if (!root) { return; }
|
| 44 |
+
var togEl = root.querySelector(".cs-tog"), scenEl = root.querySelector(".cs-scen"), gridEl = root.querySelector(".cs-grid");
|
| 45 |
+
function cssVar(n, fb) { var v = getComputedStyle(document.documentElement).getPropertyValue(n); return (v && v.trim()) || fb; }
|
| 46 |
+
var PROV = { OpenAI: "#2a8f82", Anthropic: "#a66e4e", Google: "#3d5a80", Mistral: "#c4553a" };
|
| 47 |
+
var DATA = null, SEL = 0;
|
| 48 |
+
function esc(s) { return String(s).replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">"); }
|
| 49 |
+
function bar(label, val, color) {
|
| 50 |
+
return '<div class="cs-bar"><div class="cs-blab"><span>' + esc(label) + '</span><span>' + val.toFixed(0) + '%</span></div>' +
|
| 51 |
+
'<div class="cs-track"><div class="cs-fill" style="width:' + Math.max(0, Math.min(100, val)) + '%;background:' + color + '"></div></div></div>';
|
| 52 |
+
}
|
| 53 |
+
function draw() {
|
| 54 |
+
if (!DATA) { return; }
|
| 55 |
+
var OPT = cssVar("--opt", "#e07a5f"), PES = cssVar("--pes", "#3d5a80"), MUT = cssVar("--muted-color", "#888");
|
| 56 |
+
// tabs
|
| 57 |
+
togEl.innerHTML = DATA.cases.map(function (c, i) {
|
| 58 |
+
return '<button class="cs-btn" data-i="' + i + '" aria-pressed="' + (i === SEL ? "true" : "false") + '">' + esc(c.domain) + "</button>";
|
| 59 |
+
}).join("");
|
| 60 |
+
togEl.querySelectorAll(".cs-btn").forEach(function (b) {
|
| 61 |
+
b.addEventListener("click", function () { SEL = +b.getAttribute("data-i"); draw(); });
|
| 62 |
+
});
|
| 63 |
+
var c = DATA.cases[SEL];
|
| 64 |
+
scenEl.innerHTML = '<div class="cs-dom">' + esc(c.domain) + '</div><div class="cs-txt">' + esc(c.scenario) + '</div>' +
|
| 65 |
+
'<div class="cs-qs"><span class="cs-qpos"><b>Positive:</b> ' + esc(c.pos) + '</span><span class="cs-qneg"><b>Negative:</b> ' + esc(c.neg) + '</span></div>';
|
| 66 |
+
gridEl.innerHTML = DATA.models.map(function (m) {
|
| 67 |
+
var d = c[m.name], col = PROV[m.provider] || MUT;
|
| 68 |
+
var sum = d.good + d.bad, skew = sum - 100;
|
| 69 |
+
var badgeC = skew >= 0 ? OPT : PES;
|
| 70 |
+
var badgeT = (skew >= 0 ? "+" : "−") + Math.abs(skew).toFixed(0) + " " + (skew >= 0 ? "optimistic" : "pessimistic");
|
| 71 |
+
var missTxt = skew >= 0
|
| 72 |
+
? Math.abs(skew).toFixed(0) + " pts double-counted toward the good outcome"
|
| 73 |
+
: Math.abs(skew).toFixed(0) + " pts double-counted toward the bad outcome";
|
| 74 |
+
return '<div class="cs-card">' +
|
| 75 |
+
'<div class="cs-mhead"><span class="cs-dot" style="background:' + col + '"></span><span class="cs-mname">' + esc(m.name) + '</span><span class="cs-mtag">' + esc(m.dir) + '</span></div>' +
|
| 76 |
+
bar("P(positive)", d.good, col) +
|
| 77 |
+
bar("P(negative)", d.bad, MUT) +
|
| 78 |
+
'<div class="cs-foot"><span class="cs-sum">sum ' + sum.toFixed(0) + ' · ' + esc(missTxt) + '</span>' +
|
| 79 |
+
'<span class="cs-badge" style="background:' + badgeC + '">' + badgeT + '</span></div>' +
|
| 80 |
+
'</div>';
|
| 81 |
+
}).join("");
|
| 82 |
+
}
|
| 83 |
+
fetch("data/cases.json").then(function (r) { if (!r.ok) { throw new Error("x"); } return r.json(); })
|
| 84 |
+
.then(function (j) { DATA = j; draw(); })
|
| 85 |
+
.catch(function () { scenEl.innerHTML = '<div class="cs-txt">Could not load data/cases.json.</div>'; });
|
| 86 |
+
new MutationObserver(draw).observe(document.documentElement, { attributes: true, attributeFilter: ["data-theme"] });
|
| 87 |
+
})();
|
| 88 |
+
</script>
|
app/src/content/embeds/crosslingual.html
CHANGED
|
@@ -32,7 +32,10 @@
|
|
| 32 |
var W = root.clientWidth || 680, mL = 124, mT = 22, rowH = 24;
|
| 33 |
var cw = Math.min(56, (W - mL - 10) / langs.length);
|
| 34 |
var H = mT + models.length * rowH + 6;
|
| 35 |
-
var
|
|
|
|
|
|
|
|
|
|
| 36 |
langs.forEach(function (l, j) { svg.append("text").attr("x", mL + j * cw + cw / 2).attr("y", mT - 7).attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 11).attr("font-weight", 600).text(l); });
|
| 37 |
models.forEach(function (m, i) {
|
| 38 |
var yy = mT + i * rowH;
|
|
|
|
| 32 |
var W = root.clientWidth || 680, mL = 124, mT = 22, rowH = 24;
|
| 33 |
var cw = Math.min(56, (W - mL - 10) / langs.length);
|
| 34 |
var H = mT + models.length * rowH + 6;
|
| 35 |
+
var gridW = mL + langs.length * cw;
|
| 36 |
+
var ox = Math.max(0, (W - gridW) / 2);
|
| 37 |
+
var svgEl = d3.select(wrap).html("").append("svg").attr("viewBox", "0 0 " + W + " " + H).attr("width", W).attr("height", H);
|
| 38 |
+
var svg = svgEl.append("g").attr("transform", "translate(" + ox + ",0)");
|
| 39 |
langs.forEach(function (l, j) { svg.append("text").attr("x", mL + j * cw + cw / 2).attr("y", mT - 7).attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 11).attr("font-weight", 600).text(l); });
|
| 40 |
models.forEach(function (m, i) {
|
| 41 |
var yy = mT + i * rowH;
|
app/src/content/embeds/hero-axis.html
DELETED
|
@@ -1,87 +0,0 @@
|
|
| 1 |
-
<div id="ob-hero" role="img" aria-label="An optimism axis: 16 language models placed by their directional bias. Most sit on the optimistic (warm) right side; only Anthropic's Opus and Sonnet sit on the pessimistic (cool) left side.">
|
| 2 |
-
<svg id="ob-hero-svg" width="100%"></svg>
|
| 3 |
-
</div>
|
| 4 |
-
<style>
|
| 5 |
-
#ob-hero { width: 100%; border-radius: 16px; overflow: hidden; background: radial-gradient(120% 140% at 50% 0%, #12141b 0%, #08090e 70%); border: 1px solid rgba(255,255,255,.10); padding: 6px 0 2px; }
|
| 6 |
-
#ob-hero svg { display: block; width: 100%; height: auto; }
|
| 7 |
-
#ob-hero .obh-pole { font: 600 12px/1 system-ui, sans-serif; letter-spacing: .14em; }
|
| 8 |
-
#ob-hero .obh-name { font: 11px/1 system-ui, sans-serif; }
|
| 9 |
-
</style>
|
| 10 |
-
<script>
|
| 11 |
-
(function () {
|
| 12 |
-
var root = document.getElementById("ob-hero");
|
| 13 |
-
var svg = root && window.d3 ? window.d3.select("#ob-hero-svg") : null;
|
| 14 |
-
if (!svg) { return; }
|
| 15 |
-
var d3 = window.d3;
|
| 16 |
-
var OPT = "#e8896f", PES = "#7d9fd4", MUT = "rgba(255,255,255,.55)";
|
| 17 |
-
var DATA = null;
|
| 18 |
-
|
| 19 |
-
function draw() {
|
| 20 |
-
if (!DATA) { return; }
|
| 21 |
-
var W = root.clientWidth || 900, H = Math.max(180, Math.min(240, W * 0.26));
|
| 22 |
-
var mX = 90, axisY = H * 0.56;
|
| 23 |
-
svg.attr("viewBox", "0 0 " + W + " " + H).attr("height", H).selectAll("*").remove();
|
| 24 |
-
|
| 25 |
-
var ext = d3.extent(DATA.models, function (d) { return d.skew; });
|
| 26 |
-
var x = d3.scaleLinear().domain([Math.min(-10, ext[0] - 1), ext[1] + 1]).range([mX, W - mX]);
|
| 27 |
-
|
| 28 |
-
// gradient axis bar
|
| 29 |
-
var defs = svg.append("defs");
|
| 30 |
-
var grad = defs.append("linearGradient").attr("id", "obh-grad").attr("x1", "0").attr("x2", "1");
|
| 31 |
-
grad.append("stop").attr("offset", "0%").attr("stop-color", PES);
|
| 32 |
-
grad.append("stop").attr("offset", "50%").attr("stop-color", "#6b6f7a");
|
| 33 |
-
grad.append("stop").attr("offset", "100%").attr("stop-color", OPT);
|
| 34 |
-
svg.append("rect").attr("x", mX).attr("y", axisY - 1.5).attr("width", W - 2 * mX).attr("height", 3).attr("rx", 1.5).attr("fill", "url(#obh-grad)").attr("opacity", 0.55);
|
| 35 |
-
|
| 36 |
-
// zero tick
|
| 37 |
-
svg.append("line").attr("x1", x(0)).attr("x2", x(0)).attr("y1", axisY - 26).attr("y2", axisY + 26).attr("stroke", "rgba(255,255,255,.25)").attr("stroke-dasharray", "3 3");
|
| 38 |
-
svg.append("text").attr("x", x(0)).attr("y", axisY + 40).attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 10).text("0 (coherent)");
|
| 39 |
-
|
| 40 |
-
// poles
|
| 41 |
-
svg.append("text").attr("class", "obh-pole").attr("x", mX - 8).attr("y", axisY + 4).attr("text-anchor", "end").attr("fill", PES).text("PESSIMISTIC");
|
| 42 |
-
svg.append("text").attr("class", "obh-pole").attr("x", W - mX + 8).attr("y", axisY + 4).attr("text-anchor", "start").attr("fill", OPT).text("OPTIMISTIC");
|
| 43 |
-
|
| 44 |
-
// bracket over the two Anthropic pessimists (left side)
|
| 45 |
-
var pess = DATA.models.filter(function (m) { return m.skew < 0; });
|
| 46 |
-
if (pess.length) {
|
| 47 |
-
var pxs = pess.map(function (m) { return x(m.skew); });
|
| 48 |
-
var bL = Math.min.apply(null, pxs) - 16, bR = Math.max.apply(null, pxs) + 16, bY = axisY - 56;
|
| 49 |
-
var br = svg.append("g");
|
| 50 |
-
br.append("path").attr("d", "M" + bL + "," + (bY + 8) + " L" + bL + "," + bY + " L" + bR + "," + bY + " L" + bR + "," + (bY + 8))
|
| 51 |
-
.attr("fill", "none").attr("stroke", PES).attr("stroke-width", 1.2).attr("opacity", 0.85);
|
| 52 |
-
br.append("text").attr("x", (bL + bR) / 2).attr("y", bY - 6).attr("text-anchor", "middle").attr("fill", PES).attr("font-size", 11).attr("font-weight", 600).text("only Anthropic is pessimistic");
|
| 53 |
-
}
|
| 54 |
-
|
| 55 |
-
// dots, all lifted off the axis bar so none blend into the gradient
|
| 56 |
-
var topSkew = d3.max(DATA.models, function (m) { return m.skew; });
|
| 57 |
-
var rows = DATA.models.map(function (m, i) { return { m: m, jit: (((i % 4) + 1)) * 14 * (i % 2 ? 1 : -1) }; });
|
| 58 |
-
var g = svg.append("g");
|
| 59 |
-
rows.forEach(function (r, i) {
|
| 60 |
-
var pesDot = r.m.skew < 0, c = pesDot ? PES : OPT;
|
| 61 |
-
var cy = pesDot ? axisY - 30 : axisY + r.jit; // pessimists raised together, clearly above the line
|
| 62 |
-
var rad = (pesDot ? 7 : 4.5) + 0.16 * Math.abs(r.m.skew);
|
| 63 |
-
// stem from axis to dot
|
| 64 |
-
g.append("line").attr("x1", x(r.m.skew)).attr("x2", x(r.m.skew)).attr("y1", axisY).attr("y2", cy)
|
| 65 |
-
.attr("stroke", pesDot ? PES : "rgba(255,255,255,.18)").attr("stroke-width", pesDot ? 1.2 : 0.7).attr("opacity", pesDot ? 0.7 : 0.5);
|
| 66 |
-
if (pesDot) {
|
| 67 |
-
g.append("circle").attr("cx", x(r.m.skew)).attr("cy", cy).attr("r", rad + 4).attr("fill", "none").attr("stroke", PES).attr("stroke-width", 1).attr("opacity", 0.55);
|
| 68 |
-
}
|
| 69 |
-
g.append("circle").attr("cx", x(r.m.skew)).attr("cy", cy).attr("r", rad).attr("fill", c).attr("fill-opacity", 0.96)
|
| 70 |
-
.attr("stroke", pesDot ? "#fff" : "#0c0d12").attr("stroke-width", pesDot ? 1.4 : 1);
|
| 71 |
-
var label = pesDot || r.m.skew === topSkew;
|
| 72 |
-
if (label) {
|
| 73 |
-
g.append("text").attr("class", "obh-name").attr("x", x(r.m.skew)).attr("y", pesDot ? cy + rad + 13 : cy - rad - 5).attr("text-anchor", "middle")
|
| 74 |
-
.attr("fill", pesDot ? PES : "rgba(255,255,255,.85)").attr("font-weight", pesDot ? 700 : 400)
|
| 75 |
-
.text(r.m.name + " " + (r.m.skew > 0 ? "+" : "") + r.m.skew.toFixed(1));
|
| 76 |
-
}
|
| 77 |
-
});
|
| 78 |
-
|
| 79 |
-
svg.append("text").attr("x", W / 2).attr("y", H - 8).attr("text-anchor", "middle").attr("fill", "rgba(255,255,255,.4)").attr("font-size", 10).text("16 models · 14 optimistic, 2 pessimistic · directional Skew on Track B");
|
| 80 |
-
}
|
| 81 |
-
|
| 82 |
-
fetch("data/skew_models.json").then(function (r) { if (!r.ok) { throw new Error("x"); } return r.json(); })
|
| 83 |
-
.then(function (j) { DATA = j; draw(); }).catch(function () {});
|
| 84 |
-
if (window.ResizeObserver) { var t = null; new ResizeObserver(function () { clearTimeout(t); t = setTimeout(draw, 140); }).observe(root); }
|
| 85 |
-
else { window.addEventListener("resize", draw); }
|
| 86 |
-
})();
|
| 87 |
-
</script>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
app/src/content/embeds/robustness.html
ADDED
|
@@ -0,0 +1,108 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<div id="ob-rb" role="img" aria-label="Bias-stability plane: mean Skew across six languages on the x-axis versus inter-language standard deviation on the y-axis, for 17 models. Most models sit to the right of zero (optimist) and low (robust); only Anthropic's Opus and Sonnet are robust pessimists.">
|
| 2 |
+
<div class="rb-head"><span class="rb-title">Bias-stability plane</span><span class="rb-sub">how big, and how stable across languages</span></div>
|
| 3 |
+
<div class="rb-wrap"></div>
|
| 4 |
+
<div class="rb-cap">Mean Skew across six native-prompt languages (horizontal) against the spread across those languages (vertical). Right of the line is optimist, left is pessimist; below the dashed line is language-robust. Hover for values.</div>
|
| 5 |
+
</div>
|
| 6 |
+
<style>
|
| 7 |
+
#ob-rb { font:13px/1.4 system-ui,sans-serif; color:var(--text-color,#222); width:100%; max-width:540px; margin:0 auto; position:relative; }
|
| 8 |
+
#ob-rb .rb-head { display:flex; align-items:baseline; gap:10px; margin-bottom:8px; }
|
| 9 |
+
#ob-rb .rb-title { font-weight:650; font-size:15px; }
|
| 10 |
+
#ob-rb .rb-sub { font-size:12px; color:var(--muted-color,#666); margin-left:auto; }
|
| 11 |
+
#ob-rb svg { display:block; width:100%; height:auto; overflow:visible; }
|
| 12 |
+
#ob-rb .rb-cap { margin-top:8px; font-size:12px; color:var(--muted-color,#666); }
|
| 13 |
+
#ob-rb .rb-tip { position:absolute; pointer-events:none; z-index:20; opacity:0; transform:translate(-50%,-100%); transition:opacity .1s; background:var(--surface-bg,#fff); color:var(--text-color,#222); border:1px solid var(--border-color,#ccc); border-radius:8px; padding:7px 9px; font-size:12px; line-height:1.5; box-shadow:0 6px 24px rgba(0,0,0,.18); white-space:nowrap; }
|
| 14 |
+
#ob-rb .rb-tip b { font-weight:650; }
|
| 15 |
+
</style>
|
| 16 |
+
<script>
|
| 17 |
+
(function () {
|
| 18 |
+
var root = document.getElementById("ob-rb");
|
| 19 |
+
if (!root || !window.d3) { return; }
|
| 20 |
+
var d3 = window.d3, wrap = root.querySelector(".rb-wrap");
|
| 21 |
+
var tip = document.createElement("div"); tip.className = "rb-tip"; root.appendChild(tip);
|
| 22 |
+
function cssVar(n, fb) { var v = getComputedStyle(document.documentElement).getPropertyValue(n); return (v && v.trim()) || fb; }
|
| 23 |
+
// paper PROVIDER_PALETTE (fig_alignment_stability)
|
| 24 |
+
var PROV = { OpenAI: "#2a8f82", Anthropic: "#a66e4e", Google: "#3d5a80", Mistral: "#c4553a", Alibaba: "#c49a3a", DeepSeek: "#5c6b73", Zhipu: "#8a8478", NVIDIA: "#4a4540", Meta: "#1877f2" };
|
| 25 |
+
var DATA = null;
|
| 26 |
+
function draw() {
|
| 27 |
+
if (!DATA) { return; }
|
| 28 |
+
wrap.innerHTML = "";
|
| 29 |
+
var OPT = cssVar("--opt", "#e07a5f"), PES = cssVar("--pes", "#3d5a80"),
|
| 30 |
+
MUT = cssVar("--muted-color", "#666"), TXT = cssVar("--text-color", "#222"),
|
| 31 |
+
SF = cssVar("--surface-bg", "#fff"), BD = cssVar("--border-color", "#e7e2dc");
|
| 32 |
+
var S = Math.min(root.clientWidth || 520, 520); // square
|
| 33 |
+
var m = { t: 30, b: 40, l: 30, r: 14 };
|
| 34 |
+
var W = S, H = S;
|
| 35 |
+
var XL = 18, YMAX = 4;
|
| 36 |
+
var x = d3.scaleLinear().domain([-XL, XL]).range([m.l, W - m.r]);
|
| 37 |
+
var y = d3.scaleLinear().domain([-0.3, YMAX]).range([H - m.b, m.t]);
|
| 38 |
+
var sig = DATA.models.map(function (d) { return d.sigma; }).sort(function (a, b) { return a - b; });
|
| 39 |
+
var MED = sig[Math.floor(sig.length / 2)]; // robust/volatile split
|
| 40 |
+
var svg = d3.select(wrap).append("svg").attr("viewBox", "0 0 " + W + " " + H).attr("width", W).attr("height", H);
|
| 41 |
+
|
| 42 |
+
// four quadrant tints (robust band richer)
|
| 43 |
+
function rect(x0, x1, y0, y1, c, op) { svg.append("rect").attr("x", x(x0)).attr("y", y(y1)).attr("width", x(x1) - x(x0)).attr("height", y(y0) - y(y1)).attr("fill", c).attr("opacity", op); }
|
| 44 |
+
rect(0, XL, MED, YMAX, OPT, 0.05); rect(-XL, 0, MED, YMAX, PES, 0.05);
|
| 45 |
+
rect(0, XL, -0.3, MED, OPT, 0.11); rect(-XL, 0, -0.3, MED, PES, 0.11);
|
| 46 |
+
|
| 47 |
+
// dividers
|
| 48 |
+
svg.append("line").attr("x1", x(0)).attr("x2", x(0)).attr("y1", y(-0.3)).attr("y2", y(YMAX)).attr("stroke", MUT).attr("stroke-width", 1).attr("opacity", 0.65);
|
| 49 |
+
svg.append("line").attr("x1", x(-XL)).attr("x2", x(XL)).attr("y1", y(MED)).attr("y2", y(MED)).attr("stroke", MUT).attr("stroke-width", 1).attr("stroke-dasharray", "5 4").attr("opacity", 0.5);
|
| 50 |
+
|
| 51 |
+
// quadrant labels
|
| 52 |
+
[[XL - 0.5, YMAX - 0.15, "VOLATILE OPTIMIST", OPT, "end", "hanging"],
|
| 53 |
+
[-XL + 0.5, YMAX - 0.15, "VOLATILE PESSIMIST", PES, "start", "hanging"],
|
| 54 |
+
[XL - 0.5, -0.18, "ROBUST OPTIMIST", OPT, "end", "auto"],
|
| 55 |
+
[-XL + 0.5, -0.18, "ROBUST PESSIMIST", PES, "start", "auto"]].forEach(function (q) {
|
| 56 |
+
svg.append("text").attr("x", x(q[0])).attr("y", y(q[1])).attr("text-anchor", q[4]).attr("dominant-baseline", q[5]).attr("fill", q[3]).attr("opacity", 0.82).attr("font-size", 10).attr("font-weight", 700).attr("letter-spacing", "0.02em").text(q[2]);
|
| 57 |
+
});
|
| 58 |
+
|
| 59 |
+
// y ticks
|
| 60 |
+
[0, 1, 2, 3, 4].forEach(function (t) {
|
| 61 |
+
svg.append("text").attr("x", m.l - 5).attr("y", y(t) + 3).attr("text-anchor", "end").attr("fill", MUT).attr("font-size", 9).text(t);
|
| 62 |
+
});
|
| 63 |
+
// x ticks
|
| 64 |
+
[-15, -10, -5, 0, 5, 10, 15].forEach(function (t) {
|
| 65 |
+
svg.append("text").attr("x", x(t)).attr("y", H - m.b + 15).attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 9).text(t > 0 ? "+" + t : t);
|
| 66 |
+
});
|
| 67 |
+
|
| 68 |
+
// ---- label placement with vertical declutter ----
|
| 69 |
+
var lbls = DATA.models.map(function (d) {
|
| 70 |
+
var px = x(d.skew), py = y(d.sigma);
|
| 71 |
+
var FORCE = { "Llama-3.3-70B": 1 }; // override to dodge same-σ neighbour
|
| 72 |
+
var side = FORCE[d.name] || (px > W * 0.6 ? -1 : 1); // -1 = label left, 1 = label right
|
| 73 |
+
var w = d.name.length * 5.7 + 4;
|
| 74 |
+
var lx = px + side * 9, ly = py - 2;
|
| 75 |
+
var x0 = side < 0 ? lx - w : lx, x1 = side < 0 ? lx : lx + w;
|
| 76 |
+
return { d: d, px: px, py: py, side: side, anchor: side < 0 ? "end" : "start", lx: lx, ly: ly, x0: x0, x1: x1 };
|
| 77 |
+
});
|
| 78 |
+
lbls.sort(function (a, b) { return a.ly - b.ly; });
|
| 79 |
+
for (var i = 0; i < lbls.length; i++) {
|
| 80 |
+
for (var j = 0; j < i; j++) {
|
| 81 |
+
var a = lbls[j], b = lbls[i];
|
| 82 |
+
if (b.x0 < a.x1 && a.x0 < b.x1 && Math.abs(b.ly - a.ly) < 12.5) { b.ly = a.ly + 12.5; }
|
| 83 |
+
}
|
| 84 |
+
}
|
| 85 |
+
|
| 86 |
+
// points + connectors + labels
|
| 87 |
+
lbls.forEach(function (L) {
|
| 88 |
+
var d = L.d, c = PROV[d.provider] || MUT;
|
| 89 |
+
if (Math.abs(L.ly - (L.py - 2)) > 6) {
|
| 90 |
+
svg.append("line").attr("x1", L.px).attr("y1", L.py).attr("x2", L.lx).attr("y2", L.ly - 3).attr("stroke", MUT).attr("stroke-width", 0.6).attr("opacity", 0.4);
|
| 91 |
+
}
|
| 92 |
+
svg.append("circle").attr("cx", L.px).attr("cy", L.py).attr("r", 5.5).attr("fill", c).attr("stroke", SF).attr("stroke-width", 1.3).style("cursor", "pointer")
|
| 93 |
+
.on("mousemove", function (ev) { var pr = root.getBoundingClientRect(); tip.innerHTML = "<b>" + d.name + "</b><br>mean Skew " + (d.skew > 0 ? "+" : "") + d.skew.toFixed(1) + " pp<br>inter-language σ " + d.sigma.toFixed(2) + " pp"; tip.style.left = (ev.clientX - pr.left) + "px"; tip.style.top = (ev.clientY - pr.top - 8) + "px"; tip.style.opacity = 1; })
|
| 94 |
+
.on("mouseleave", function () { tip.style.opacity = 0; });
|
| 95 |
+
svg.append("text").attr("x", L.lx).attr("y", L.ly).attr("text-anchor", L.anchor).attr("fill", TXT).attr("font-size", 9.5).attr("font-weight", 500).text(d.name);
|
| 96 |
+
});
|
| 97 |
+
|
| 98 |
+
// axis labels
|
| 99 |
+
svg.append("text").attr("x", m.l).attr("y", m.t - 14).attr("text-anchor", "start").attr("fill", MUT).attr("font-size", 10.5).text("Mean Skew across 6 languages (pp) →");
|
| 100 |
+
svg.append("text").attr("transform", "translate(13," + ((m.t + H - m.b) / 2) + ")rotate(-90)").attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 10.5).text("Inter-language σ (pp)");
|
| 101 |
+
}
|
| 102 |
+
fetch("data/robustness.json").then(function (r) { if (!r.ok) { throw new Error("x"); } return r.json(); })
|
| 103 |
+
.then(function (j) { DATA = j; draw(); })
|
| 104 |
+
.catch(function () { wrap.innerHTML = '<div class="rb-cap">Could not load data/robustness.json.</div>'; });
|
| 105 |
+
if (window.ResizeObserver) { var t = null; new ResizeObserver(function () { clearTimeout(t); t = setTimeout(draw, 130); }).observe(root); } else { window.addEventListener("resize", draw); }
|
| 106 |
+
new MutationObserver(draw).observe(document.documentElement, { attributes: true, attributeFilter: ["data-theme"] });
|
| 107 |
+
})();
|
| 108 |
+
</script>
|
app/src/content/embeds/skew-forest.html
CHANGED
|
@@ -58,7 +58,12 @@
|
|
| 58 |
text: cssVar("--text-color", "#222"), muted: cssVar("--muted-color", "#666"),
|
| 59 |
border: cssVar("--border-color", "#ccc"), surface: cssVar("--surface-bg", "#fff") };
|
| 60 |
|
| 61 |
-
var
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
if (MODE === "skew") { rows.sort(function (a, b) { return b.skew - a.skew; }); }
|
| 63 |
else {
|
| 64 |
var top = {}; DATA.models.forEach(function (m) { if (!(m.provider in top)) top[m.provider] = d3.max(DATA.models.filter(function (x) { return x.provider === m.provider; }), function (x) { return x.skew; }); });
|
|
|
|
| 58 |
text: cssVar("--text-color", "#222"), muted: cssVar("--muted-color", "#666"),
|
| 59 |
border: cssVar("--border-color", "#ccc"), surface: cssVar("--surface-bg", "#fff") };
|
| 60 |
|
| 61 |
+
var n = DATA.n || 60;
|
| 62 |
+
var rows = DATA.models.map(function (m) {
|
| 63 |
+
var h = 1.96 * m.sigma / Math.sqrt(n);
|
| 64 |
+
return { name: m.name, provider: m.provider, size: m.size, dir: m.dir,
|
| 65 |
+
skew: m.skew, sigma: m.sigma, lo: m.skew - h, hi: m.skew + h };
|
| 66 |
+
});
|
| 67 |
if (MODE === "skew") { rows.sort(function (a, b) { return b.skew - a.skew; }); }
|
| 68 |
else {
|
| 69 |
var top = {}; DATA.models.forEach(function (m) { if (!(m.provider in top)) top[m.provider] = d3.max(DATA.models.filter(function (x) { return x.provider === m.provider; }), function (x) { return x.skew; }); });
|
app/src/content/embeds/valence-quadrant.html
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
<div id="ob-vq" role="img" aria-label="Good-side push versus bad-side push for six models. Optimistic models inflate both axes; Sonnet's pessimism is one-sided, underestimating good outcomes while keeping bad-outcome estimates near accurate."></div>
|
| 2 |
<style>
|
| 3 |
-
#ob-vq { font: 13px/1.4 system-ui, sans-serif; color: var(--text-color,#222); width:100%; max-width:
|
| 4 |
-
#ob-vq svg { display:block; width:100%; height:auto; }
|
| 5 |
#ob-vq .vq-tip { position:absolute; pointer-events:none; z-index:20; opacity:0; transform:translate(-50%,-100%); transition:opacity .1s; background:var(--surface-bg,#fff); color:var(--text-color,#222); border:1px solid var(--border-color,#ccc); border-radius:8px; padding:7px 9px; font-size:12px; line-height:1.5; box-shadow:0 6px 24px rgba(0,0,0,.18); white-space:nowrap; }
|
| 6 |
#ob-vq .vq-tip b { font-weight:650; }
|
| 7 |
</style>
|
|
@@ -13,37 +13,56 @@
|
|
| 13 |
var svg = d3.select(root).append("svg");
|
| 14 |
var tip = document.createElement("div"); tip.className = "vq-tip"; root.appendChild(tip);
|
| 15 |
function cssVar(n, fb) { var v = getComputedStyle(document.documentElement).getPropertyValue(n); return (v && v.trim()) || fb; }
|
| 16 |
-
var PROV = { OpenAI: "#10a37f", Anthropic: "#d97757", Google: "#4285f4" };
|
| 17 |
var DATA = null;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 18 |
function draw() {
|
| 19 |
if (!DATA) { return; }
|
| 20 |
-
var OPT = cssVar("--opt", "#e07a5f"), PES = cssVar("--pes", "#3d5a80"),
|
| 21 |
-
|
|
|
|
|
|
|
| 22 |
var x = d3.scaleLinear().domain([-LIM, LIM]).range([m, W - m]);
|
| 23 |
var y = d3.scaleLinear().domain([-LIM, LIM]).range([H - m, m]);
|
| 24 |
svg.attr("viewBox", "0 0 " + W + " " + H).attr("height", H).selectAll("*").remove();
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 25 |
// quadrant labels
|
| 26 |
-
[[7.5,
|
| 27 |
-
|
|
|
|
| 28 |
});
|
|
|
|
|
|
|
| 29 |
svg.append("line").attr("x1", x(-LIM)).attr("x2", x(LIM)).attr("y1", y(0)).attr("y2", y(0)).attr("stroke", BD);
|
| 30 |
svg.append("line").attr("x1", x(0)).attr("x2", x(0)).attr("y1", y(-LIM)).attr("y2", y(LIM)).attr("stroke", BD);
|
| 31 |
-
|
| 32 |
-
var pos = {}; DATA.models.forEach(function (
|
| 33 |
var defs = svg.append("defs");
|
| 34 |
DATA.pairs.forEach(function (p, i) {
|
| 35 |
-
var
|
| 36 |
-
defs.append("marker").attr("id", "vqa" + i).attr("viewBox", "0 0 10 10").attr("refX", 8).attr("refY", 5).attr("markerWidth", 6).attr("markerHeight", 6).attr("orient", "auto").append("path").attr("d", "M0,0L10,5L0,10").attr("fill", c).attr("opacity", 0.
|
| 37 |
-
svg.append("line").attr("x1", pos[p[0]][0]).attr("y1", pos[p[0]][1]).attr("x2", pos[p[1]][0]).attr("y2", pos[p[1]][1]).attr("stroke", c).attr("stroke-width", 1
|
| 38 |
});
|
|
|
|
|
|
|
| 39 |
DATA.models.forEach(function (d) {
|
| 40 |
-
var
|
| 41 |
-
g
|
| 42 |
-
g.append("
|
| 43 |
-
|
|
|
|
|
|
|
| 44 |
});
|
| 45 |
-
|
| 46 |
-
svg.append("text").attr("
|
|
|
|
| 47 |
}
|
| 48 |
fetch("data/valence.json").then(function (r) { if (!r.ok) { throw new Error("x"); } return r.json(); }).then(function (j) { DATA = j; draw(); }).catch(function () {});
|
| 49 |
if (window.ResizeObserver) { var t = null; new ResizeObserver(function () { clearTimeout(t); t = setTimeout(draw, 130); }).observe(root); } else { window.addEventListener("resize", draw); }
|
|
|
|
| 1 |
<div id="ob-vq" role="img" aria-label="Good-side push versus bad-side push for six models. Optimistic models inflate both axes; Sonnet's pessimism is one-sided, underestimating good outcomes while keeping bad-outcome estimates near accurate."></div>
|
| 2 |
<style>
|
| 3 |
+
#ob-vq { font: 13px/1.4 system-ui, sans-serif; color: var(--text-color,#222); width:100%; max-width:480px; margin:0 auto; position:relative; }
|
| 4 |
+
#ob-vq svg { display:block; width:100%; height:auto; overflow:visible; }
|
| 5 |
#ob-vq .vq-tip { position:absolute; pointer-events:none; z-index:20; opacity:0; transform:translate(-50%,-100%); transition:opacity .1s; background:var(--surface-bg,#fff); color:var(--text-color,#222); border:1px solid var(--border-color,#ccc); border-radius:8px; padding:7px 9px; font-size:12px; line-height:1.5; box-shadow:0 6px 24px rgba(0,0,0,.18); white-space:nowrap; }
|
| 6 |
#ob-vq .vq-tip b { font-weight:650; }
|
| 7 |
</style>
|
|
|
|
| 13 |
var svg = d3.select(root).append("svg");
|
| 14 |
var tip = document.createElement("div"); tip.className = "vq-tip"; root.appendChild(tip);
|
| 15 |
function cssVar(n, fb) { var v = getComputedStyle(document.documentElement).getPropertyValue(n); return (v && v.trim()) || fb; }
|
|
|
|
| 16 |
var DATA = null;
|
| 17 |
+
// hand-tuned label placement to avoid overlap in the top cluster
|
| 18 |
+
var LBL = {
|
| 19 |
+
"GPT-5.4-mini": [9, -9, "start"], "GPT-5.4": [0, -11, "middle"], "Haiku 4.5": [0, 16, "middle"],
|
| 20 |
+
"Flash 3": [9, 13, "start"], "Pro 3.1": [0, -11, "middle"], "Sonnet 4.6": [10, 4, "start"]
|
| 21 |
+
};
|
| 22 |
function draw() {
|
| 23 |
if (!DATA) { return; }
|
| 24 |
+
var OPT = cssVar("--opt", "#e07a5f"), PES = cssVar("--pes", "#3d5a80"),
|
| 25 |
+
MUT = cssVar("--muted-color", "#666"), TXT = cssVar("--text-color", "#222"),
|
| 26 |
+
BD = cssVar("--border-color", "#ccc"), SF = cssVar("--surface-bg", "#fff");
|
| 27 |
+
var W = Math.min(460, root.clientWidth || 460), H = W, m = 50, LIM = 11;
|
| 28 |
var x = d3.scaleLinear().domain([-LIM, LIM]).range([m, W - m]);
|
| 29 |
var y = d3.scaleLinear().domain([-LIM, LIM]).range([H - m, m]);
|
| 30 |
svg.attr("viewBox", "0 0 " + W + " " + H).attr("height", H).selectAll("*").remove();
|
| 31 |
+
|
| 32 |
+
// faint quadrant tints (optimism quadrant warm, pessimism cool)
|
| 33 |
+
function tint(qx, qy, c) { svg.append("rect").attr("x", x(qx)).attr("y", y(qy + LIM)).attr("width", (W - 2 * m) / 2).attr("height", (H - 2 * m) / 2).attr("fill", c).attr("opacity", 0.05); }
|
| 34 |
+
tint(0, 0, OPT); tint(-LIM, -LIM, PES);
|
| 35 |
+
|
| 36 |
// quadrant labels
|
| 37 |
+
[[7.5, 9.4, "OVERCLAIM", "#7d6440"], [-7.5, -9.4, "UNDERCLAIM", "#6b6358"],
|
| 38 |
+
[-7.5, 9.4, "PESSIMISM", PES], [7.5, -9.4, "OPTIMISM", OPT]].forEach(function (q) {
|
| 39 |
+
svg.append("text").attr("x", x(q[0])).attr("y", y(q[1])).attr("text-anchor", "middle").attr("fill", q[3]).attr("opacity", 0.7).attr("font-size", 10).attr("font-weight", 700).attr("letter-spacing", "0.03em").text(q[2]);
|
| 40 |
});
|
| 41 |
+
|
| 42 |
+
// axes only (no diagonal)
|
| 43 |
svg.append("line").attr("x1", x(-LIM)).attr("x2", x(LIM)).attr("y1", y(0)).attr("y2", y(0)).attr("stroke", BD);
|
| 44 |
svg.append("line").attr("x1", x(0)).attr("x2", x(0)).attr("y1", y(-LIM)).attr("y2", y(LIM)).attr("stroke", BD);
|
| 45 |
+
|
| 46 |
+
var pos = {}; DATA.models.forEach(function (mm) { pos[mm.name] = [x(mm.good), y(mm.bad)]; });
|
| 47 |
var defs = svg.append("defs");
|
| 48 |
DATA.pairs.forEach(function (p, i) {
|
| 49 |
+
var c = MUT;
|
| 50 |
+
defs.append("marker").attr("id", "vqa" + i).attr("viewBox", "0 0 10 10").attr("refX", 8).attr("refY", 5).attr("markerWidth", 6).attr("markerHeight", 6).attr("orient", "auto").append("path").attr("d", "M0,0L10,5L0,10").attr("fill", c).attr("opacity", 0.4);
|
| 51 |
+
svg.append("line").attr("x1", pos[p[0]][0]).attr("y1", pos[p[0]][1]).attr("x2", pos[p[1]][0]).attr("y2", pos[p[1]][1]).attr("stroke", c).attr("stroke-width", 1).attr("stroke-dasharray", "3 3").attr("opacity", 0.35).attr("marker-end", "url(#vqa" + i + ")");
|
| 52 |
});
|
| 53 |
+
|
| 54 |
+
var PROV = { OpenAI: "#2a8f82", Anthropic: "#a66e4e", Google: "#3d5a80", Mistral: "#c4553a" };
|
| 55 |
DATA.models.forEach(function (d) {
|
| 56 |
+
var skew = d.good + d.bad, c = PROV[d.provider] || MUT;
|
| 57 |
+
var g = svg.append("g").style("cursor", "pointer");
|
| 58 |
+
g.append("circle").attr("cx", x(d.good)).attr("cy", y(d.bad)).attr("r", 7).attr("fill", c).attr("stroke", SF).attr("stroke-width", 1.6);
|
| 59 |
+
var L = LBL[d.name] || [10, 4, "start"];
|
| 60 |
+
g.append("text").attr("x", x(d.good) + L[0]).attr("y", y(d.bad) + L[1]).attr("text-anchor", L[2]).attr("fill", TXT).attr("font-size", 11).attr("font-weight", 500).text(d.name);
|
| 61 |
+
g.on("mousemove", function (ev) { var pr = root.getBoundingClientRect(); tip.innerHTML = "<b>" + d.name + "</b><br>good-side push: " + (d.good > 0 ? "+" : "") + d.good + "<br>bad-side push: " + (d.bad > 0 ? "+" : "") + d.bad + "<br>Skew " + (skew > 0 ? "+" : "") + skew.toFixed(1); tip.style.left = (ev.clientX - pr.left) + "px"; tip.style.top = (ev.clientY - pr.top - 8) + "px"; tip.style.opacity = 1; }).on("mouseleave", function () { tip.style.opacity = 0; });
|
| 62 |
});
|
| 63 |
+
|
| 64 |
+
svg.append("text").attr("x", (m + W - m) / 2).attr("y", H - 10).attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 11).text("good-side push P(good) − 50");
|
| 65 |
+
svg.append("text").attr("transform", "translate(13," + H / 2 + ") rotate(-90)").attr("text-anchor", "middle").attr("fill", MUT).attr("font-size", 11).text("bad-side push P(bad) − 50");
|
| 66 |
}
|
| 67 |
fetch("data/valence.json").then(function (r) { if (!r.ok) { throw new Error("x"); } return r.json(); }).then(function (j) { DATA = j; draw(); }).catch(function () {});
|
| 68 |
if (window.ResizeObserver) { var t = null; new ResizeObserver(function () { clearTimeout(t); t = setTimeout(draw, 130); }).observe(root); } else { window.addEventListener("resize", draw); }
|