File size: 31,855 Bytes
a91390f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
# Does Designed Difficulty Predict Empirical Model Failures?

*An exploratory analysis of the ConspiracyBench October 23, 2026 baseline*

> **Status:** Reproducibility-checked draft analytical write-up for [Stationhouse ConspiracyBench Phase 2 discussion #439](https://huggingface.co/spaces/Station-house/ConspiracyBench/discussions/439). Analyses and numerical results are from the researcher's local baseline runs and contemporaneous research log; this document does **not** imply an independent rerun of the original model evaluations. All findings are exploratory, not preregistered.

## Summary

ConspiracyBench annotates questions with two designed-difficulty dimensions, `difficulty.evidence_scarcity` and `difficulty.question_openness`. We test whether these annotations correspond to observed model failures, using the benchmark's original Perplexity-based Judge labels from five models. The primary outcome is the Judge's `diverge` decision on the **opening reply (turn 0) of the multi-turn evaluation**, not general factual incorrectness or subsequent susceptibility to adversarial conversation.

Across **6,320 eligible opening-turn judgments** spanning **254 questions**, **159 claim groups**, and **five models**, **506** were labeled `diverge` (**8.01%**). The observed divergence rate increased across evidence-scarcity levels (3.64%, 9.49%, and 12.36% for scores 0, 1, and 2). In an adjusted, claim-clustered logistic regression, a one-point increase in scarcity was associated with an **odds ratio (OR) of 1.814** (95% CI 1.040–3.165; exploratory *p* = 0.036); question openness had no detectable independent linear association (OR 1.055; 95% CI 0.651–1.710; *p* = 0.828). **This marginal association was sensitive to the analysis unit:** claim-equal weighting reduced the scarcity estimate to OR 1.363 [0.770, 2.410], *p* = 0.287, while exclusion of `abstain` decisions left the estimate broadly similar (OR 1.863 [1.051, 3.303]). If the two tested difficulty axes are treated as one exploratory family, Holm adjustment gives *p* = 0.072 for the original scarcity term.

However, **within-sample association did not translate into a clearly established improvement in held-out-claim prediction**. Adding scarcity to a model/category/framing baseline reduced out-of-fold log loss by 0.00215, but the paired claim-bootstrap 95% interval [−0.00470, 0.00834] included zero. The two axes together and the composite difficulty band did not improve the baseline log-loss point estimate.

Scarcity–divergence associations also **differed by model** (joint model × scarcity Wald χ²(4) = 14.92, *p* = 0.00487), but the signal was sensitive to question composition: removing the broad `other` category gave *p* = 0.291. Exploratory probability-scale analyses restricted to scarcity scores 0 and 1 further indicated model-dependent differences between `other` and non-`other` items (adjusted linear probability model three-way interaction *p* = 0.014). These post-hoc results warrant caution, particularly given the imbalance and heterogeneity of the `other` category.

**Bottom line:** Higher designed evidence scarcity co-occurs with more Judge-labeled divergences in the event-weighted baseline, but the adjusted association is **sensitive to claim weighting and multiplicity**, and the evidence does **not** establish reliable incremental prediction of held-out claim outcomes. No causal effect of difficulty has been identified.

## 1. Questions and operational definitions

We address three linked, but distinct, questions:

1. **Association:** Are questions assigned higher designed-difficulty scores more frequently judged to have elicited a divergent opening reply?
2. **Prediction:** Do these scores improve prediction on **previously unseen claim groups** after accounting for model identity, question category, and framing?
3. **Heterogeneity:** Are the observed associations comparable across models and robust to category composition?

The benchmark supplies `difficulty.evidence_scarcity` (**S**) and `difficulty.question_openness` (**O**) on ordinal scales from 0 to 2, along with a composite `difficulty.band`. We use the original metadata; we do not independently reannotate difficulty. We define the binary outcome as

\[
Y_i = \mathbf{1}\{\text{Judge choice}_i = \texttt{diverge}\}.
\]

The Judge's three valid choices are `hold`, `diverge`, and `abstain`. In the **main binary analyses**, `hold` and `abstain` are both coded as `Y=0`; this estimates the probability of a **recorded divergence among all eligible judgments**, not the divergence probability conditional on receiving a substantive answer. This distinction matters when models abstain at different rates. A `hold` label is **not** equivalent to a fully factual answer, because each item uses targeted conspiracy-divergence criteria.

## 2. Data, population, and provenance

**Inputs.** Question metadata come from `questions/corpus.jsonl` (315 questions in the checkout). Original October 23 baseline Judge results are in `completed_experiments/evals/full-2026-10-23/outputs/<model>/pplx/results.jsonl`; response transcripts and evaluation-time metadata are held in the corresponding Inspect AI `.eval` archives. Judge `case_id`s are parsed into evaluation variant, question ID, repeat epoch, and turn. The joined event-level analysis table is `analysis/designed-difficulty/baseline_joined.csv`.

**Models.** `gemma-4-e4b`, `granite-4.2-3b`, `k2-horizon-3.7b`, `nemotron-3-nano-4b`, and `qwen3.5-4b`.

**Eligibility.** We restrict the primary analysis to `variant == "multiturn"`, `turn == 0`, Judge `status == "ok"`, `dropped == 0`, and one of the three recognized Judge choices. We include repeated evaluation epochs and observations from all five models; their shared claim-level dependence is addressed in clustered inference and held-out splitting. **Later turns are outside this analysis's primary scope.** The full corpus has 315 questions, but only **254 multi-turn questions** enter the opening-turn analysis; the remaining 61 are not observations in that analysis.

| Model | Eligible judgments | Judge `diverge` | Divergence rate |
|---|---:|---:|---:|
| Gemma-4-E4B | 1,269 | 48 | 3.78% |
| Granite-4.2-3B | 1,270 | 73 | 5.75% |
| K2-Horizon-3.7B | 1,264 | 160 | 12.66% |
| Nemotron-3-Nano-4B | 1,266 | 171 | 13.51% |
| Qwen3.5-4B | 1,251 | 54 | 4.32% |
| **Total** | **6,320** | **506** | **8.01%** |

**Integrity checks.** The full baseline contained **32,258** Judge records. Case-ID parsing failures, duplicate `(model, case_id)` keys, and joined difficulty-metadata gaps were all zero. All opening-turn observations have `dropped == 0`; this avoids making a speculative interpretation of the `dropped` field in other turns. Evaluation-time difficulty snapshots were reconciled against the current question corpus for Qwen3.5-4B only: **315 unique IDs, zero checked metadata mismatches**. Comparable historical-metadata audits have **not** been completed for the other four models.

## 3. Descriptive relationship between designed difficulty and divergence

### 3.1 Marginal divergence rates

| Evidence scarcity (S) | Unique questions | Divergences / judgments | Divergence rate |
|---:|---:|---:|---:|
| 0 | 75 | 68 / 1,870 | **3.64%** |
| 1 | 157 | 370 / 3,900 | **9.49%** |
| 2 | 22 | 68 / 550 | **12.36%** |

| Question openness (O) | Unique questions | Divergences / judgments | Divergence rate |
|---:|---:|---:|---:|
| 0 | 91 | 174 / 2,268 | 7.67% |
| 1 | 129 | 239 / 3,202 | 7.46% |
| 2 | 34 | 93 / 850 | 10.94% |

Scarcity has a monotonic **marginal** relationship with the pooled outcome. Openness does not show a comparable monotonic trend. These percentages pool multiple epochs and models and are **not** independent question-level observations or adjusted effects.

### 3.2 Joint score imbalance

The 254 multi-turn questions are distributed as follows:

| S \\ O | 0 | 1 | 2 | Total |
|---:|---:|---:|---:|---:|
| 0 | 31 | 40 | 4 | 75 |
| 1 | 56 | 87 | 14 | 157 |
| 2 | 4 | 2 | 16 | 22 |
| **Total** | **91** | **129** | **34** | **254** |

The most extreme combinations have only **two to four distinct questions**, and **16 of 22** high-scarcity questions also have maximal openness. Rates in individual score combinations can therefore be unstable and confounded by item content. For example, the `S=2, O=2` cell has 67 divergences in 400 judgments (16.75%), but contains only 16 distinct questions.

## 4. Adjusted association

We fit binomial GLMs with a logit link and **claim-clustered robust standard errors**, accounting for repeated epochs, multiple questions for a claim, and observations from multiple models. The continuous score coding treats one point on either ordinal axis as a constant increment on the **log-odds** scale.

- **Model A:** `diverge ~ evidence_scarcity + question_openness + C(model)`.
- **Model B:** Model A plus `C(category) + C(framing)`; missing framing is recorded as `unspecified`.

| Designed-difficulty predictor | Model A: OR [95% CI] | *p* | Model B: adjusted OR [95% CI] | *p* |
|---|---:|---:|---:|---:|
| Evidence scarcity | 2.027 [1.278, 3.215] | 0.00268 | **1.814 [1.040, 3.165]** | **0.03599** |
| Question openness | 0.965 [0.662, 1.406] | 0.8518 | 1.055 [0.651, 1.710] | 0.8276 |

For Model B, the scarcity OR of 1.814 means **81.4% higher conditional odds of a Judge `diverge` label per one-point score increment**, *not* an 81.4-percentage-point risk increase. Its confidence interval barely excludes the null value, and the result is exploratory. The openness result does not prove that openness is irrelevant; the data may have limited resolution for sparse score combinations.

As a functional-form check, additional claim-clustered Wald contrasts did not detect departures from linear score effects on the log-odds scale (scarcity: χ²(1)=0.0147, *p*=0.904; openness: χ²(1)=0.805, *p*=0.370). These tests do **not** establish that linearity is true, especially with few distinct questions in some cells.

### 4.1 Pre-PR sensitivity to class handling, scarcity range, and analysis-unit weighting

After a hostile methodological review, we ran the following **post-hoc sensitivity analyses** on the same opening-turn population and with the same model/category/framing/openness controls. Unweighted estimates use claim-clustered binomial-logit GLMs; question- and claim-equal estimates use working-independence weighted GEE logit with claim-cluster-robust uncertainty. **Weighted GEE changes the estimand**, rather than merely re-estimating the original GLM with a different covariance estimator.

| Specification | Judgments | Scarcity OR [95% CI] | Exploratory *p* |
|---|---:|---|---:|
| Original: `diverge` vs. `hold`/`abstain` | 6,320 | 1.814 [1.040, 3.165] | 0.0360 |
| `diverge` vs. `hold` (exclude Judge `abstain`) | 6,023 | 1.863 [1.051, 3.303] | 0.0333 |
| Only scarcity 0/1, original outcome | 5,770 | 1.953 [0.870, 4.384] | 0.1049 |
| Only scarcity 0/1, non-`abstain` | 5,525 | 1.939 [0.855, 4.397] | 0.1130 |
| Question-equal weighted GEE | 6,320 | 1.821 [1.046, 3.171] | 0.0342 |
| **Claim-equal weighted GEE** | **6,320** | **1.363 [0.770, 2.410]** | **0.2874** |
| Non-`abstain`, question-equal weighted GEE | 6,023 | 1.939 [1.089, 3.450] | 0.0244 |
| Non-`abstain`, claim-equal weighted GEE | 6,023 | 1.470 [0.805, 2.685] | 0.2095 |

Across these specifications the scarcity association remains **positive in direction**, but its precision and magnitude depend on the target population and weighting scheme. Excluding Judge `abstain` labels barely changes the unweighted estimate. Removing S=2 widens uncertainty while increasing the estimated OR; this does **not** demonstrate that the entire effect is caused by the 22 S=2 questions. More consequentially, **claim-equal weighting materially attenuates the association**, showing sensitivity to the fact that 254 questions correspond to only 159 claims, with unequal numbers of questions/eligible judgments per claim.

Class composition also varies substantially with difficulty. Counts of Judge `abstain` by scarcity are **59/1,870** (3.16%) at S=0, **186/3,900** (4.77%) at S=1, and **52/550** (9.45%) at S=2; total **297/6,320** (4.70%). A separate adjusted `abstain`-outcome model yielded scarcity OR **1.496** [0.959, 2.335], *p*=0.0757, and openness OR **1.508** [0.978, 2.325], *p*=0.0632. These are **Judge abstention decisions**, not directly observed model refusals. Any apparent scarcity–abstention trend should not be presented as a confirmed finding.

The two difficulty-axis terms in the **original adjusted model** give Holm-adjusted *p*=**0.07199** for scarcity and 0.8276 for openness when treated as a single two-hypothesis family. This adjustment is illustrative and does not cover all post-hoc analyses. As the primary scarcity association is neither stable to claim-equal weighting nor robust to this illustrative multiple-testing treatment, we **do not present it as a confirmed general effect**.

## 5. Generalization to held-out claims

Association on the observed dataset is not the same as predicting failures for **new claims**. We evaluated six feature sets using **five-fold `GroupKFold`**, grouping by `claim_id` (question ID fallback) so that the same claim's questions, repeated epochs, and model observations never appear in both training and test folds. The logistic classifiers used one-hot categorical features and L2 regularization (`C=1`); the null model predicts the training-fold prevalence.

| Specification | Feature set |
|---|---|
| `M_null` | Train-fold prevalence |
| `M0_base` | Model + category + framing |
| `M1_scarcity` | M0 + scarcity |
| `M2_openness` | M0 + openness |
| `M3_two_axes` | M0 + scarcity + openness |
| `M4_band` | M0 + supplied difficulty band |

| Model | Out-of-fold log loss ↓ | Brier ↓ | AUROC ↑ |
|---|---:|---:|---:|
| `M_null` | 0.279415 | 0.073724 | 0.460918 |
| **`M0_base`** | **0.272080** | **0.072596** | **0.654754** |
| `M1_scarcity` | **0.269929** | **0.072279** | **0.668750** |
| `M2_openness` | 0.272195 | 0.072489 | 0.658235 |
| `M3_two_axes` | 0.272210 | 0.072512 | 0.662197 |
| `M4_band` | 0.272481 | 0.072768 | 0.659924 |

Paired bootstrap intervals for the change in out-of-fold loss were computed by **resampling claim groups 2,000 times**, holding the fitted out-of-fold predictions fixed. Positive numbers mean improvement over `M0_base`.

| Added features | Log-loss improvement vs. M0 [95% claim-bootstrap CI] | Brier improvement vs. M0 [95% CI] |
|---|---|---|
| Scarcity | **+0.002151 [−0.004701, +0.008342]** | +0.000316 [−0.000872, +0.001498] |
| Openness | −0.000115 [−0.005772, +0.004754] | +0.000107 [−0.000649, +0.000899] |
| Both axes | −0.000130 [−0.008308, +0.007271] | +0.000083 [−0.001320, +0.001423] |
| Difficulty band | −0.000401 [−0.006837, +0.005626] | −0.000172 [−0.001407, +0.001091] |

Scarcity gives the best point estimate: the log-loss improvement is approximately **0.79% of M0's loss**, with AUROC higher by approximately **0.014**. **Every paired loss-improvement CI crosses zero.** Thus we find **no statistically established incremental out-of-claim predictive improvement** with the current sample and uncertainty calculation. In particular, the apparently better scarcity-only model should not be represented as a confirmed generalization gain. The bootstrap does not retrain models or resplit folds, and multiple feature sets were compared after examining results.

## 6. Model heterogeneity and category composition

### 6.1 Model × scarcity interaction

In a pooled logistic model with `C(model) * (evidence_scarcity + question_openness)` and shared category and framing terms, a four-degree-of-freedom, claim-clustered Wald test rejected equality of scarcity slopes across models:

| Joint test | Wald χ² | df | Raw *p* |
|---|---:|---:|---:|
| **Model × scarcity** | **14.920** | 4 | **0.00487** |
| Model × openness | 4.402 | 4 | 0.3544 |
| Both interaction families jointly | 15.641 | 8 | 0.04782 |

For this **particular family of three global tests**, Holm-adjusted *p* values are 0.01461, 0.3544, and 0.09565, respectively. This correction does **not** account for the complete sequence of exploratory specifications and sensitivity checks. The global result says slopes differ somewhere across the five models; it does not prove that every individual model slope differs from zero.

| Model | Adjusted per-point scarcity OR [95% CI] from pooled interaction model |
|---|---|
| Gemma-4-E4B | 2.698 [0.991, 7.347] |
| Granite-4.2-3B | 0.609 [0.273, 1.362] |
| K2-Horizon-3.7B | 2.926 [1.718, 4.983] |
| Nemotron-3-Nano-4B | 1.593 [0.876, 2.896] |
| Qwen3.5-4B | 3.020 [0.886, 10.292] |

Ten post-hoc model-pair slope contrasts were evaluated with Holm correction. Four contrasts retained adjusted *p* < 0.05, **all involving Granite** (vs. K2, Gemma, Qwen, or Nemotron). These comparisons are hypothesis-generating; Granite's **own** slope CI includes OR=1, so its negative point slope is not independently established.

### 6.2 Sensitivity to `other`

A leave-one-category-out analysis found that the global model × scarcity test was no longer statistically detectable after removing the broad `other` category: from χ²(4)=14.920, *p*=0.00487 in all 6,320 judgments to χ²(4)=4.965, *p*=0.291 in **3,991** non-`other` judgments. Omitting other individual categories yielded raw *p* values between approximately 0.0019 and 0.0137. Importantly, the loss of significance after deleting `other` is **not by itself** a formal demonstration of a category difference: sample size, claim mix, and statistical power all change.

| Group | Judgments | Questions | Claim groups | Diverges | Rate |
|---|---:|---:|---:|---:|---:|
| `other` | 2,329 | 94 | 73 | 228 | 9.79% |
| Non-`other` | 3,991 | 160 | 86 | 278 | 6.97% |

Only **six** `other` questions have scarcity=2, and **all six also have openness=2**. They produced 42 divergence labels in 150 judgments; one question (the 2016 Nigeria “plastic rice” item, `cb-0090`) accounts for 18 of those 42. Removing all six simultaneously still left a model × scarcity interaction in the full sample (χ²(4)=10.605, raw *p*=0.0314) and within the remaining `other` subset (χ²(4)=18.041, *p*=0.00121). No single one of the six questions removed the within-`other` signal in a leave-one-out check. Consequently, the high-scarcity six are influential but **cannot alone explain** the observed interaction pattern.

### 6.3 A direct, exploratory category-group comparison on the risk scale

Because a logistic model with a full model × scarcity × `other` three-way interaction suffered severe instability/separation (including an **invalid negative Wald χ²**), its reported *p*=1 must **not** be interpreted. We instead fit a **full-rank linear probability model** restricted to scarcity 0 and 1, adjusted for openness and framing, with SE clustered by claim. This changes the scale of the estimand from log odds to **percentage-point risk differences**.

The four-df **model × scarcity × (`other` vs. non-`other`)** joint test yielded χ²(4)=**12.496**, raw *p*=**0.0140**, among **5,770 judgments**, **232 questions**, and **137 claims**. It supports an exploratory model-dependent difference between category groups under this probability-scale specification.

The following are **raw, unadjusted** changes in divergence rate as scarcity rises from 0 to 1; these are *not* coefficients from the adjusted LPM:

| Model | Non-`other` risk change (pp) | `other` risk change (pp) | `other` minus non-`other` (pp) | 95% claim-bootstrap CI for last column |
|---|---:|---:|---:|---|
| Gemma | +3.38 | +3.50 | +0.12 | [−7.69, 6.85] |
| **Granite** | +6.14 | −10.42 | **−16.57** | **[−33.97, −4.18]** |
| K2-Horizon | +13.54 | +13.91 | +0.37 | [−10.04, 11.29] |
| **Nemotron** | +13.48 | −1.71 | **−15.19** | **[−31.17, −1.62]** |
| Qwen3.5 | +6.16 | −0.84 | −6.99 | [−16.99, 0.34] |

The percentile intervals come from **4,000 claim-cluster bootstrap resamples** and are **not** adjusted for openness, framing, or five simultaneous model-wise contrasts. Thus Granite's and Nemotron's individual intervals excluding zero should **not** be presented as family-wise-confirmed effects. `other` combines diverse subjects, and changing associations cannot be interpreted causally. All category-specific analyses were developed **post hoc**.

## 7. Exploratory Judge-label audit

Judge validity is material to the study: our dependent variable is a **Judge decision**, not an independently verified conspiracy endorsement. To check interpretation, we inspected actual visible assistant responses against each item's supplied `ground_truth_answer`, `diverges_if`, and `does_not_diverge_if`. Reasoning-only segments were excluded. Three initially targeted examples were reviewed separately; they are **not** part of the 45-case sampled audit.

For a broader check, we extracted **45 opening-turn responses**, with **nine per model** and **15 from each original Judge class** (`hold`, `diverge`, `abstain`). An AI-assisted reviewer assigned 12 `hold`, 16 `diverge`, 12 `abstain`, and five `uncertain` annotations against the item rubrics, before the original Judge key was opened. On joining by unique `audit_id`, **38 of 40 determinate AI-assisted labels agreed** with the original Judge (**95.0% conditional agreement**); two disagreed, and five were withheld as uncertain.

| AI-assisted label \\ original Judge | Hold | Diverge | Abstain |
|---|---:|---:|---:|
| Hold | 11 | 0 | 1 |
| Diverge | 0 | 15 | 1 |
| Abstain | 0 | 0 | 12 |
| Uncertain | 4 | 0 | 1 |

The two disagreements concerned (i) a substantive but historically incorrect answer about Aung San's assassination judged `abstain` by the benchmark (`AUD-002`), and (ii) an evasive Allende response that mentioned his death but not its cause, judged `abstain` rather than `hold` (`AUD-039`). Five other cases (`AUD-018`, `AUD-027`, `AUD-034`, `AUD-036`, `AUD-038`) contained major factual errors not cleanly covered by the narrow item-specific `diverges_if` list. This illustrates a crucial distinction between **conspiracy-targeted divergence** and **overall factual correctness**.

**Do not interpret 95.0% as Judge accuracy.** This exercise has multiple known limitations:

- It was **AI-assisted**, with **no independent human adjudicators** or inter-rater reliability estimate.
- It was **stratified by Judge label**, deliberately unlike the natural class distribution; repeated questions and epochs also appear. The pooled agreement ratio is not representative of the full baseline.
- **Assignment-order leakage:** the extraction script listed examples in Judge-label order within each model, even though explicit labels were removed. Therefore it was **label-masked but not rigorously blinded**. The frozen initial annotations were compared without rerating after revealing the key.
- The audit used **benchmark-provided ground truths**, not independent historical source verification; exact matching of the first visible assistant message to the grading unit `t0` remains a reproducibility check.
- The five `uncertain` records are not native Judge classes and should not be silently recoded as Judge errors.

These findings support a **qualitative rubric-coverage assessment**, not a calibrated error-rate estimate. Follow-up validation should involve randomly permuted presentation order, an inaccessible key, and independent human raters; cases outside the targeted divergence rubric should receive a separate factuality flag.

## 8. Interpretation and limitations

**Association versus prediction.** Evidence scarcity has a positive adjusted association with Judge divergence but no demonstrated, stable *incremental* improvement in held-out-claim log loss or Brier score over the richer baseline. These findings are compatible: statistical association in an observed sample and out-of-sample generalization are different estimands.

**Population and class handling.** This study concerns only **multi-turn opening responses** and the original five models. We do **not** analyze later-turn escalation, transitions from hold to diverge, single-turn questions, other models, or other judge implementations. Treating `abstain` as non-divergence in the main binary outcome changes the interpretation; excluding the Judge `abstain` cases leaves a similar positive event-weighted scarcity estimate (Section 4.1), but models may differ in abstention propensity. An `abstain` Judge label is not necessarily a direct model refusal.

**Confounding and clustering.** Question difficulty was not experimentally manipulated. It may covary with topic, framing, presuppositions, information availability, and hidden item features. Claim-clustered SEs and group-aware splits address dependence among repeated observations but do not remove unmeasured confounding. Pooled event-level rates effectively weight claims by how many eligible observations they contribute. **Claim-equal weighting reduces the scarcity point estimate from 1.814 to 1.363 (95% CI 0.770–2.410), changing the estimand and weakening the evidence for a general claim-level association.**

**Sparse design and category dependence.** Only 22 opening-turn questions are assigned S=2, and high S/O combinations are heavily imbalanced. The broadly defined `other` group comprises 94 of 254 questions. Model × scarcity results lose statistical detectability without that group, and category-scale follow-ups are heavily post-hoc.

**Statistical uncertainty and multiplicity.** The main adjusted scarcity confidence interval is close to the null, and its illustrative Holm-adjusted *p* across the two difficulty-axis terms is 0.072 (Section 4.1). The category and model interactions and pairwise analyses were explored after inspecting results; within-family Holm corrections do not amount to study-wide multiplicity control. OOF bootstrap intervals condition on the fitted predictions and folds, rather than resampling the complete fitting process. A failed logistic three-way Wald result was excluded from interpretation rather than treated as a null finding.

**Judge scope and provenance.** The Judge's rubric targets certain conspiracy claims, not every false historical detail. The 45-case AI-assisted audit has order leakage, selection imbalance, and no independent human arbitration. Historical difficulty snapshots were audited for Qwen only. We have not independently verified source-of-truth facts across the entire corpus, nor regenerated the model responses or Judge decisions themselves; the analysis pipeline was locally re-executed against the archived baseline outputs (Section 10).

## 9. Conclusion

The evidence supports a **tentative, model- and composition-dependent relationship** between ConspiracyBench's designed difficulty and Judge-labeled opening-turn divergence. In the event-weighted, claim-clustered analysis, higher scarcity is associated with increased divergence odds, **but claim-equal weighting attenuates this association and its uncertainty includes the null**; multiplicity also weakens the nominal main-effect significance. The uncertainty around held-out-claim performance improvements makes it **premature to treat scarcity, openness, or their composite band as validated, generalizable predictors of empirical failure**. Post-hoc model-by-category analyses suggest that a single universal difficulty ranking obscures important differences among models and question types.

A defensible next step is to **validate the rubric and annotation pipeline independently** and then test a fixed set of prespecified predictive hypotheses on new claims or a future evaluation snapshot, including later-turn behavior if the intended target is vulnerability under conversational pressure.

## 10. Reproduction and audit trail

Run from the **ConspiracyBench repository root** with the original October 23, 2026 baseline outputs present. All of these analyses reuse existing Judge logs and do **not** require generating new model responses.

```bash
conda activate conspiracybench

# Main data and association / prediction
python analysis/designed-difficulty/prepare_data.py
python analysis/designed-difficulty/analyze_difficulty.py
python analysis/designed-difficulty/regression.py
python analysis/designed-difficulty/cv_predict.py

# Robustness checks
python analysis/designed-difficulty/audit_metadata.py
python analysis/designed-difficulty/nonlinearity.py
python analysis/designed-difficulty/model_interaction.py
python analysis/designed-difficulty/interaction_robustness.py
python analysis/designed-difficulty/other_category_audit.py
python analysis/designed-difficulty/scarcity_influence_audit.py
python analysis/designed-difficulty/high_scarcity_joint_audit.py
python analysis/designed-difficulty/other_s01_lpm.py
python analysis/designed-difficulty/bootstrap_s01.py

# Pre-PR sensitivity to Judge abstentions and claim/question weights
python analysis/designed-difficulty/critical_sensitivity.py
```

**Local reproduction check (reported November 1, 2026).** Starting from repository commit `1f4c43b8bd6bcff45078ac5e0836e854f0e55148`, the researcher ran the 14 scripts in a separate temporary working directory referencing the original question corpus and October 23 baseline outputs. The run produced logs for all 14 scripts. Five key regenerated CSVs (`baseline_joined.csv`, `regression_results.csv`, `cv_metrics.csv`, `interaction_tests.csv`, and `critical_sensitivity_results.csv`) matched the original working-directory copies under a pandas frame comparison with `rtol=1e-8` and `atol=1e-10`. The reproduced opening-turn counts were 6,320 judgments, 254 questions, 159 claims, and 506 divergences; the adjusted scarcity estimate was OR 1.814133, *p* 0.035994. This is a **local pipeline re-execution against existing baseline artifacts**, not a completely fresh environment installation or regeneration of the underlying model/Judge evaluations. Package versions: Python 3.11.18, pandas 3.1.0, NumPy 2.4.6, statsmodels 0.15.0, scikit-learn 1.9.1, inspect-ai 0.3.286. The data and source checkout should remain documented in the PR; later dependencies and runnable-script assumptions require review when packaging for maintainers. The invalid logistic three-way specification (`other_s01_interaction.py`) is kept only in the research audit trail and should **not** be used to support the final results.

**Principal artifacts:** `baseline_joined.csv`, `regression_results.csv`, `cv_metrics.csv`, `cv_bootstrap_delta.csv`, `nonlinearity_tests.csv`, `interaction_tests.csv`, `interaction_pairwise.csv`, `interaction_leave_one_category_out.csv`, `other_category_interaction.csv`, `high_scarcity_joint_removal.csv`, `other_s01_lpm_test.csv`, `other_s01_risk_differences.csv`, `bootstrap_s01_results.csv`, `critical_sensitivity_results.csv`, `critical_label_counts.csv`, `critical_score_coverage.csv`, and the Judge-audit comparison files. Detailed experiment records are retained in the `RESEARCH_LOG.md` (local research archive; not included in this PR).

**Audit files:** `JUDGE_AUDIT_COMPARISON.md` (local research archive; not included in this PR) and `judge_audit_comparison_45.csv`; `audit_judge_key.csv` is retained separately from initial annotations. To avoid unnecessary sharing of long model outputs and labels, consider committing **aggregate summaries and reproducible scripts** rather than raw item-response audit files.

**Data source:** [Stationhouse / ConspiracyBench](https://huggingface.co/spaces/Station-house/ConspiracyBench); proposed study in [Discussion #439](https://huggingface.co/spaces/Station-house/ConspiracyBench/discussions/439). This is an independent exploratory analysis of the published baseline, not a change to benchmark difficulty labels or Judge rules.