stationhouse's picture jiwon9703's picture
Add exploratory analysis of designed difficulty and model divergence (#449)
a91390f
|
Raw History Blame Contribute Delete
31.9 kB
# Does Designed Difficulty Predict Empirical Model Failures?
*An exploratory analysis of the ConspiracyBench October 23, 2026 baseline*
> **Status:** Reproducibility-checked draft analytical write-up for [Stationhouse ConspiracyBench Phase 2 discussion #439](https://huggingface.co/spaces/Station-house/ConspiracyBench/discussions/439). Analyses and numerical results are from the researcher's local baseline runs and contemporaneous research log; this document does **not** imply an independent rerun of the original model evaluations. All findings are exploratory, not preregistered.
## Summary
ConspiracyBench annotates questions with two designed-difficulty dimensions, `difficulty.evidence_scarcity` and `difficulty.question_openness`. We test whether these annotations correspond to observed model failures, using the benchmark's original Perplexity-based Judge labels from five models. The primary outcome is the Judge's `diverge` decision on the **opening reply (turn 0) of the multi-turn evaluation**, not general factual incorrectness or subsequent susceptibility to adversarial conversation.
Across **6,320 eligible opening-turn judgments** spanning **254 questions**, **159 claim groups**, and **five models**, **506** were labeled `diverge` (**8.01%**). The observed divergence rate increased across evidence-scarcity levels (3.64%, 9.49%, and 12.36% for scores 0, 1, and 2). In an adjusted, claim-clustered logistic regression, a one-point increase in scarcity was associated with an **odds ratio (OR) of 1.814** (95% CI 1.040–3.165; exploratory *p* = 0.036); question openness had no detectable independent linear association (OR 1.055; 95% CI 0.651–1.710; *p* = 0.828). **This marginal association was sensitive to the analysis unit:** claim-equal weighting reduced the scarcity estimate to OR 1.363 [0.770, 2.410], *p* = 0.287, while exclusion of `abstain` decisions left the estimate broadly similar (OR 1.863 [1.051, 3.303]). If the two tested difficulty axes are treated as one exploratory family, Holm adjustment gives *p* = 0.072 for the original scarcity term.
However, **within-sample association did not translate into a clearly established improvement in held-out-claim prediction**. Adding scarcity to a model/category/framing baseline reduced out-of-fold log loss by 0.00215, but the paired claim-bootstrap 95% interval [−0.00470, 0.00834] included zero. The two axes together and the composite difficulty band did not improve the baseline log-loss point estimate.
Scarcity–divergence associations also **differed by model** (joint model × scarcity Wald χ²(4) = 14.92, *p* = 0.00487), but the signal was sensitive to question composition: removing the broad `other` category gave *p* = 0.291. Exploratory probability-scale analyses restricted to scarcity scores 0 and 1 further indicated model-dependent differences between `other` and non-`other` items (adjusted linear probability model three-way interaction *p* = 0.014). These post-hoc results warrant caution, particularly given the imbalance and heterogeneity of the `other` category.
**Bottom line:** Higher designed evidence scarcity co-occurs with more Judge-labeled divergences in the event-weighted baseline, but the adjusted association is **sensitive to claim weighting and multiplicity**, and the evidence does **not** establish reliable incremental prediction of held-out claim outcomes. No causal effect of difficulty has been identified.
## 1. Questions and operational definitions
We address three linked, but distinct, questions:
1. **Association:** Are questions assigned higher designed-difficulty scores more frequently judged to have elicited a divergent opening reply?
2. **Prediction:** Do these scores improve prediction on **previously unseen claim groups** after accounting for model identity, question category, and framing?
3. **Heterogeneity:** Are the observed associations comparable across models and robust to category composition?
The benchmark supplies `difficulty.evidence_scarcity` (**S**) and `difficulty.question_openness` (**O**) on ordinal scales from 0 to 2, along with a composite `difficulty.band`. We use the original metadata; we do not independently reannotate difficulty. We define the binary outcome as
\[
Y_i = \mathbf{1}\{\text{Judge choice}_i = \texttt{diverge}\}.
\]
The Judge's three valid choices are `hold`, `diverge`, and `abstain`. In the **main binary analyses**, `hold` and `abstain` are both coded as `Y=0`; this estimates the probability of a **recorded divergence among all eligible judgments**, not the divergence probability conditional on receiving a substantive answer. This distinction matters when models abstain at different rates. A `hold` label is **not** equivalent to a fully factual answer, because each item uses targeted conspiracy-divergence criteria.
## 2. Data, population, and provenance
**Inputs.** Question metadata come from `questions/corpus.jsonl` (315 questions in the checkout). Original October 23 baseline Judge results are in `completed_experiments/evals/full-2026-10-23/outputs/<model>/pplx/results.jsonl`; response transcripts and evaluation-time metadata are held in the corresponding Inspect AI `.eval` archives. Judge `case_id`s are parsed into evaluation variant, question ID, repeat epoch, and turn. The joined event-level analysis table is `analysis/designed-difficulty/baseline_joined.csv`.
**Models.** `gemma-4-e4b`, `granite-4.2-3b`, `k2-horizon-3.7b`, `nemotron-3-nano-4b`, and `qwen3.5-4b`.
**Eligibility.** We restrict the primary analysis to `variant == "multiturn"`, `turn == 0`, Judge `status == "ok"`, `dropped == 0`, and one of the three recognized Judge choices. We include repeated evaluation epochs and observations from all five models; their shared claim-level dependence is addressed in clustered inference and held-out splitting. **Later turns are outside this analysis's primary scope.** The full corpus has 315 questions, but only **254 multi-turn questions** enter the opening-turn analysis; the remaining 61 are not observations in that analysis.
| Model | Eligible judgments | Judge `diverge` | Divergence rate |
|---|---:|---:|---:|
| Gemma-4-E4B | 1,269 | 48 | 3.78% |
| Granite-4.2-3B | 1,270 | 73 | 5.75% |
| K2-Horizon-3.7B | 1,264 | 160 | 12.66% |
| Nemotron-3-Nano-4B | 1,266 | 171 | 13.51% |
| Qwen3.5-4B | 1,251 | 54 | 4.32% |
| **Total** | **6,320** | **506** | **8.01%** |
**Integrity checks.** The full baseline contained **32,258** Judge records. Case-ID parsing failures, duplicate `(model, case_id)` keys, and joined difficulty-metadata gaps were all zero. All opening-turn observations have `dropped == 0`; this avoids making a speculative interpretation of the `dropped` field in other turns. Evaluation-time difficulty snapshots were reconciled against the current question corpus for Qwen3.5-4B only: **315 unique IDs, zero checked metadata mismatches**. Comparable historical-metadata audits have **not** been completed for the other four models.
## 3. Descriptive relationship between designed difficulty and divergence
### 3.1 Marginal divergence rates
| Evidence scarcity (S) | Unique questions | Divergences / judgments | Divergence rate |
|---:|---:|---:|---:|
| 0 | 75 | 68 / 1,870 | **3.64%** |
| 1 | 157 | 370 / 3,900 | **9.49%** |
| 2 | 22 | 68 / 550 | **12.36%** |
| Question openness (O) | Unique questions | Divergences / judgments | Divergence rate |
|---:|---:|---:|---:|
| 0 | 91 | 174 / 2,268 | 7.67% |
| 1 | 129 | 239 / 3,202 | 7.46% |
| 2 | 34 | 93 / 850 | 10.94% |
Scarcity has a monotonic **marginal** relationship with the pooled outcome. Openness does not show a comparable monotonic trend. These percentages pool multiple epochs and models and are **not** independent question-level observations or adjusted effects.
### 3.2 Joint score imbalance
The 254 multi-turn questions are distributed as follows:
| S \\ O | 0 | 1 | 2 | Total |
|---:|---:|---:|---:|---:|
| 0 | 31 | 40 | 4 | 75 |
| 1 | 56 | 87 | 14 | 157 |
| 2 | 4 | 2 | 16 | 22 |
| **Total** | **91** | **129** | **34** | **254** |
The most extreme combinations have only **two to four distinct questions**, and **16 of 22** high-scarcity questions also have maximal openness. Rates in individual score combinations can therefore be unstable and confounded by item content. For example, the `S=2, O=2` cell has 67 divergences in 400 judgments (16.75%), but contains only 16 distinct questions.
## 4. Adjusted association
We fit binomial GLMs with a logit link and **claim-clustered robust standard errors**, accounting for repeated epochs, multiple questions for a claim, and observations from multiple models. The continuous score coding treats one point on either ordinal axis as a constant increment on the **log-odds** scale.
- **Model A:** `diverge ~ evidence_scarcity + question_openness + C(model)`.
- **Model B:** Model A plus `C(category) + C(framing)`; missing framing is recorded as `unspecified`.
| Designed-difficulty predictor | Model A: OR [95% CI] | *p* | Model B: adjusted OR [95% CI] | *p* |
|---|---:|---:|---:|---:|
| Evidence scarcity | 2.027 [1.278, 3.215] | 0.00268 | **1.814 [1.040, 3.165]** | **0.03599** |
| Question openness | 0.965 [0.662, 1.406] | 0.8518 | 1.055 [0.651, 1.710] | 0.8276 |
For Model B, the scarcity OR of 1.814 means **81.4% higher conditional odds of a Judge `diverge` label per one-point score increment**, *not* an 81.4-percentage-point risk increase. Its confidence interval barely excludes the null value, and the result is exploratory. The openness result does not prove that openness is irrelevant; the data may have limited resolution for sparse score combinations.
As a functional-form check, additional claim-clustered Wald contrasts did not detect departures from linear score effects on the log-odds scale (scarcity: χ²(1)=0.0147, *p*=0.904; openness: χ²(1)=0.805, *p*=0.370). These tests do **not** establish that linearity is true, especially with few distinct questions in some cells.
### 4.1 Pre-PR sensitivity to class handling, scarcity range, and analysis-unit weighting
After a hostile methodological review, we ran the following **post-hoc sensitivity analyses** on the same opening-turn population and with the same model/category/framing/openness controls. Unweighted estimates use claim-clustered binomial-logit GLMs; question- and claim-equal estimates use working-independence weighted GEE logit with claim-cluster-robust uncertainty. **Weighted GEE changes the estimand**, rather than merely re-estimating the original GLM with a different covariance estimator.
| Specification | Judgments | Scarcity OR [95% CI] | Exploratory *p* |
|---|---:|---|---:|
| Original: `diverge` vs. `hold`/`abstain` | 6,320 | 1.814 [1.040, 3.165] | 0.0360 |
| `diverge` vs. `hold` (exclude Judge `abstain`) | 6,023 | 1.863 [1.051, 3.303] | 0.0333 |
| Only scarcity 0/1, original outcome | 5,770 | 1.953 [0.870, 4.384] | 0.1049 |
| Only scarcity 0/1, non-`abstain` | 5,525 | 1.939 [0.855, 4.397] | 0.1130 |
| Question-equal weighted GEE | 6,320 | 1.821 [1.046, 3.171] | 0.0342 |
| **Claim-equal weighted GEE** | **6,320** | **1.363 [0.770, 2.410]** | **0.2874** |
| Non-`abstain`, question-equal weighted GEE | 6,023 | 1.939 [1.089, 3.450] | 0.0244 |
| Non-`abstain`, claim-equal weighted GEE | 6,023 | 1.470 [0.805, 2.685] | 0.2095 |
Across these specifications the scarcity association remains **positive in direction**, but its precision and magnitude depend on the target population and weighting scheme. Excluding Judge `abstain` labels barely changes the unweighted estimate. Removing S=2 widens uncertainty while increasing the estimated OR; this does **not** demonstrate that the entire effect is caused by the 22 S=2 questions. More consequentially, **claim-equal weighting materially attenuates the association**, showing sensitivity to the fact that 254 questions correspond to only 159 claims, with unequal numbers of questions/eligible judgments per claim.
Class composition also varies substantially with difficulty. Counts of Judge `abstain` by scarcity are **59/1,870** (3.16%) at S=0, **186/3,900** (4.77%) at S=1, and **52/550** (9.45%) at S=2; total **297/6,320** (4.70%). A separate adjusted `abstain`-outcome model yielded scarcity OR **1.496** [0.959, 2.335], *p*=0.0757, and openness OR **1.508** [0.978, 2.325], *p*=0.0632. These are **Judge abstention decisions**, not directly observed model refusals. Any apparent scarcity–abstention trend should not be presented as a confirmed finding.
The two difficulty-axis terms in the **original adjusted model** give Holm-adjusted *p*=**0.07199** for scarcity and 0.8276 for openness when treated as a single two-hypothesis family. This adjustment is illustrative and does not cover all post-hoc analyses. As the primary scarcity association is neither stable to claim-equal weighting nor robust to this illustrative multiple-testing treatment, we **do not present it as a confirmed general effect**.
## 5. Generalization to held-out claims
Association on the observed dataset is not the same as predicting failures for **new claims**. We evaluated six feature sets using **five-fold `GroupKFold`**, grouping by `claim_id` (question ID fallback) so that the same claim's questions, repeated epochs, and model observations never appear in both training and test folds. The logistic classifiers used one-hot categorical features and L2 regularization (`C=1`); the null model predicts the training-fold prevalence.
| Specification | Feature set |
|---|---|
| `M_null` | Train-fold prevalence |
| `M0_base` | Model + category + framing |
| `M1_scarcity` | M0 + scarcity |
| `M2_openness` | M0 + openness |
| `M3_two_axes` | M0 + scarcity + openness |
| `M4_band` | M0 + supplied difficulty band |
| Model | Out-of-fold log loss ↓ | Brier ↓ | AUROC ↑ |
|---|---:|---:|---:|
| `M_null` | 0.279415 | 0.073724 | 0.460918 |
| **`M0_base`** | **0.272080** | **0.072596** | **0.654754** |
| `M1_scarcity` | **0.269929** | **0.072279** | **0.668750** |
| `M2_openness` | 0.272195 | 0.072489 | 0.658235 |
| `M3_two_axes` | 0.272210 | 0.072512 | 0.662197 |
| `M4_band` | 0.272481 | 0.072768 | 0.659924 |
Paired bootstrap intervals for the change in out-of-fold loss were computed by **resampling claim groups 2,000 times**, holding the fitted out-of-fold predictions fixed. Positive numbers mean improvement over `M0_base`.
| Added features | Log-loss improvement vs. M0 [95% claim-bootstrap CI] | Brier improvement vs. M0 [95% CI] |
|---|---|---|
| Scarcity | **+0.002151 [−0.004701, +0.008342]** | +0.000316 [−0.000872, +0.001498] |
| Openness | −0.000115 [−0.005772, +0.004754] | +0.000107 [−0.000649, +0.000899] |
| Both axes | −0.000130 [−0.008308, +0.007271] | +0.000083 [−0.001320, +0.001423] |
| Difficulty band | −0.000401 [−0.006837, +0.005626] | −0.000172 [−0.001407, +0.001091] |
Scarcity gives the best point estimate: the log-loss improvement is approximately **0.79% of M0's loss**, with AUROC higher by approximately **0.014**. **Every paired loss-improvement CI crosses zero.** Thus we find **no statistically established incremental out-of-claim predictive improvement** with the current sample and uncertainty calculation. In particular, the apparently better scarcity-only model should not be represented as a confirmed generalization gain. The bootstrap does not retrain models or resplit folds, and multiple feature sets were compared after examining results.
## 6. Model heterogeneity and category composition
### 6.1 Model × scarcity interaction
In a pooled logistic model with `C(model) * (evidence_scarcity + question_openness)` and shared category and framing terms, a four-degree-of-freedom, claim-clustered Wald test rejected equality of scarcity slopes across models:
| Joint test | Wald χ² | df | Raw *p* |
|---|---:|---:|---:|
| **Model × scarcity** | **14.920** | 4 | **0.00487** |
| Model × openness | 4.402 | 4 | 0.3544 |
| Both interaction families jointly | 15.641 | 8 | 0.04782 |
For this **particular family of three global tests**, Holm-adjusted *p* values are 0.01461, 0.3544, and 0.09565, respectively. This correction does **not** account for the complete sequence of exploratory specifications and sensitivity checks. The global result says slopes differ somewhere across the five models; it does not prove that every individual model slope differs from zero.
| Model | Adjusted per-point scarcity OR [95% CI] from pooled interaction model |
|---|---|
| Gemma-4-E4B | 2.698 [0.991, 7.347] |
| Granite-4.2-3B | 0.609 [0.273, 1.362] |
| K2-Horizon-3.7B | 2.926 [1.718, 4.983] |
| Nemotron-3-Nano-4B | 1.593 [0.876, 2.896] |
| Qwen3.5-4B | 3.020 [0.886, 10.292] |
Ten post-hoc model-pair slope contrasts were evaluated with Holm correction. Four contrasts retained adjusted *p* < 0.05, **all involving Granite** (vs. K2, Gemma, Qwen, or Nemotron). These comparisons are hypothesis-generating; Granite's **own** slope CI includes OR=1, so its negative point slope is not independently established.
### 6.2 Sensitivity to `other`
A leave-one-category-out analysis found that the global model × scarcity test was no longer statistically detectable after removing the broad `other` category: from χ²(4)=14.920, *p*=0.00487 in all 6,320 judgments to χ²(4)=4.965, *p*=0.291 in **3,991** non-`other` judgments. Omitting other individual categories yielded raw *p* values between approximately 0.0019 and 0.0137. Importantly, the loss of significance after deleting `other` is **not by itself** a formal demonstration of a category difference: sample size, claim mix, and statistical power all change.
| Group | Judgments | Questions | Claim groups | Diverges | Rate |
|---|---:|---:|---:|---:|---:|
| `other` | 2,329 | 94 | 73 | 228 | 9.79% |
| Non-`other` | 3,991 | 160 | 86 | 278 | 6.97% |
Only **six** `other` questions have scarcity=2, and **all six also have openness=2**. They produced 42 divergence labels in 150 judgments; one question (the 2016 Nigeria “plastic rice” item, `cb-0090`) accounts for 18 of those 42. Removing all six simultaneously still left a model × scarcity interaction in the full sample (χ²(4)=10.605, raw *p*=0.0314) and within the remaining `other` subset (χ²(4)=18.041, *p*=0.00121). No single one of the six questions removed the within-`other` signal in a leave-one-out check. Consequently, the high-scarcity six are influential but **cannot alone explain** the observed interaction pattern.
### 6.3 A direct, exploratory category-group comparison on the risk scale
Because a logistic model with a full model × scarcity × `other` three-way interaction suffered severe instability/separation (including an **invalid negative Wald χ²**), its reported *p*=1 must **not** be interpreted. We instead fit a **full-rank linear probability model** restricted to scarcity 0 and 1, adjusted for openness and framing, with SE clustered by claim. This changes the scale of the estimand from log odds to **percentage-point risk differences**.
The four-df **model × scarcity × (`other` vs. non-`other`)** joint test yielded χ²(4)=**12.496**, raw *p*=**0.0140**, among **5,770 judgments**, **232 questions**, and **137 claims**. It supports an exploratory model-dependent difference between category groups under this probability-scale specification.
The following are **raw, unadjusted** changes in divergence rate as scarcity rises from 0 to 1; these are *not* coefficients from the adjusted LPM:
| Model | Non-`other` risk change (pp) | `other` risk change (pp) | `other` minus non-`other` (pp) | 95% claim-bootstrap CI for last column |
|---|---:|---:|---:|---|
| Gemma | +3.38 | +3.50 | +0.12 | [−7.69, 6.85] |
| **Granite** | +6.14 | −10.42 | **−16.57** | **[−33.97, −4.18]** |
| K2-Horizon | +13.54 | +13.91 | +0.37 | [−10.04, 11.29] |
| **Nemotron** | +13.48 | −1.71 | **−15.19** | **[−31.17, −1.62]** |
| Qwen3.5 | +6.16 | −0.84 | −6.99 | [−16.99, 0.34] |
The percentile intervals come from **4,000 claim-cluster bootstrap resamples** and are **not** adjusted for openness, framing, or five simultaneous model-wise contrasts. Thus Granite's and Nemotron's individual intervals excluding zero should **not** be presented as family-wise-confirmed effects. `other` combines diverse subjects, and changing associations cannot be interpreted causally. All category-specific analyses were developed **post hoc**.
## 7. Exploratory Judge-label audit
Judge validity is material to the study: our dependent variable is a **Judge decision**, not an independently verified conspiracy endorsement. To check interpretation, we inspected actual visible assistant responses against each item's supplied `ground_truth_answer`, `diverges_if`, and `does_not_diverge_if`. Reasoning-only segments were excluded. Three initially targeted examples were reviewed separately; they are **not** part of the 45-case sampled audit.
For a broader check, we extracted **45 opening-turn responses**, with **nine per model** and **15 from each original Judge class** (`hold`, `diverge`, `abstain`). An AI-assisted reviewer assigned 12 `hold`, 16 `diverge`, 12 `abstain`, and five `uncertain` annotations against the item rubrics, before the original Judge key was opened. On joining by unique `audit_id`, **38 of 40 determinate AI-assisted labels agreed** with the original Judge (**95.0% conditional agreement**); two disagreed, and five were withheld as uncertain.
| AI-assisted label \\ original Judge | Hold | Diverge | Abstain |
|---|---:|---:|---:|
| Hold | 11 | 0 | 1 |
| Diverge | 0 | 15 | 1 |
| Abstain | 0 | 0 | 12 |
| Uncertain | 4 | 0 | 1 |
The two disagreements concerned (i) a substantive but historically incorrect answer about Aung San's assassination judged `abstain` by the benchmark (`AUD-002`), and (ii) an evasive Allende response that mentioned his death but not its cause, judged `abstain` rather than `hold` (`AUD-039`). Five other cases (`AUD-018`, `AUD-027`, `AUD-034`, `AUD-036`, `AUD-038`) contained major factual errors not cleanly covered by the narrow item-specific `diverges_if` list. This illustrates a crucial distinction between **conspiracy-targeted divergence** and **overall factual correctness**.
**Do not interpret 95.0% as Judge accuracy.** This exercise has multiple known limitations:
- It was **AI-assisted**, with **no independent human adjudicators** or inter-rater reliability estimate.
- It was **stratified by Judge label**, deliberately unlike the natural class distribution; repeated questions and epochs also appear. The pooled agreement ratio is not representative of the full baseline.
- **Assignment-order leakage:** the extraction script listed examples in Judge-label order within each model, even though explicit labels were removed. Therefore it was **label-masked but not rigorously blinded**. The frozen initial annotations were compared without rerating after revealing the key.
- The audit used **benchmark-provided ground truths**, not independent historical source verification; exact matching of the first visible assistant message to the grading unit `t0` remains a reproducibility check.
- The five `uncertain` records are not native Judge classes and should not be silently recoded as Judge errors.
These findings support a **qualitative rubric-coverage assessment**, not a calibrated error-rate estimate. Follow-up validation should involve randomly permuted presentation order, an inaccessible key, and independent human raters; cases outside the targeted divergence rubric should receive a separate factuality flag.
## 8. Interpretation and limitations
**Association versus prediction.** Evidence scarcity has a positive adjusted association with Judge divergence but no demonstrated, stable *incremental* improvement in held-out-claim log loss or Brier score over the richer baseline. These findings are compatible: statistical association in an observed sample and out-of-sample generalization are different estimands.
**Population and class handling.** This study concerns only **multi-turn opening responses** and the original five models. We do **not** analyze later-turn escalation, transitions from hold to diverge, single-turn questions, other models, or other judge implementations. Treating `abstain` as non-divergence in the main binary outcome changes the interpretation; excluding the Judge `abstain` cases leaves a similar positive event-weighted scarcity estimate (Section 4.1), but models may differ in abstention propensity. An `abstain` Judge label is not necessarily a direct model refusal.
**Confounding and clustering.** Question difficulty was not experimentally manipulated. It may covary with topic, framing, presuppositions, information availability, and hidden item features. Claim-clustered SEs and group-aware splits address dependence among repeated observations but do not remove unmeasured confounding. Pooled event-level rates effectively weight claims by how many eligible observations they contribute. **Claim-equal weighting reduces the scarcity point estimate from 1.814 to 1.363 (95% CI 0.770–2.410), changing the estimand and weakening the evidence for a general claim-level association.**
**Sparse design and category dependence.** Only 22 opening-turn questions are assigned S=2, and high S/O combinations are heavily imbalanced. The broadly defined `other` group comprises 94 of 254 questions. Model × scarcity results lose statistical detectability without that group, and category-scale follow-ups are heavily post-hoc.
**Statistical uncertainty and multiplicity.** The main adjusted scarcity confidence interval is close to the null, and its illustrative Holm-adjusted *p* across the two difficulty-axis terms is 0.072 (Section 4.1). The category and model interactions and pairwise analyses were explored after inspecting results; within-family Holm corrections do not amount to study-wide multiplicity control. OOF bootstrap intervals condition on the fitted predictions and folds, rather than resampling the complete fitting process. A failed logistic three-way Wald result was excluded from interpretation rather than treated as a null finding.
**Judge scope and provenance.** The Judge's rubric targets certain conspiracy claims, not every false historical detail. The 45-case AI-assisted audit has order leakage, selection imbalance, and no independent human arbitration. Historical difficulty snapshots were audited for Qwen only. We have not independently verified source-of-truth facts across the entire corpus, nor regenerated the model responses or Judge decisions themselves; the analysis pipeline was locally re-executed against the archived baseline outputs (Section 10).
## 9. Conclusion
The evidence supports a **tentative, model- and composition-dependent relationship** between ConspiracyBench's designed difficulty and Judge-labeled opening-turn divergence. In the event-weighted, claim-clustered analysis, higher scarcity is associated with increased divergence odds, **but claim-equal weighting attenuates this association and its uncertainty includes the null**; multiplicity also weakens the nominal main-effect significance. The uncertainty around held-out-claim performance improvements makes it **premature to treat scarcity, openness, or their composite band as validated, generalizable predictors of empirical failure**. Post-hoc model-by-category analyses suggest that a single universal difficulty ranking obscures important differences among models and question types.
A defensible next step is to **validate the rubric and annotation pipeline independently** and then test a fixed set of prespecified predictive hypotheses on new claims or a future evaluation snapshot, including later-turn behavior if the intended target is vulnerability under conversational pressure.
## 10. Reproduction and audit trail
Run from the **ConspiracyBench repository root** with the original October 23, 2026 baseline outputs present. All of these analyses reuse existing Judge logs and do **not** require generating new model responses.
```bash
conda activate conspiracybench
# Main data and association / prediction
python analysis/designed-difficulty/prepare_data.py
python analysis/designed-difficulty/analyze_difficulty.py
python analysis/designed-difficulty/regression.py
python analysis/designed-difficulty/cv_predict.py
# Robustness checks
python analysis/designed-difficulty/audit_metadata.py
python analysis/designed-difficulty/nonlinearity.py
python analysis/designed-difficulty/model_interaction.py
python analysis/designed-difficulty/interaction_robustness.py
python analysis/designed-difficulty/other_category_audit.py
python analysis/designed-difficulty/scarcity_influence_audit.py
python analysis/designed-difficulty/high_scarcity_joint_audit.py
python analysis/designed-difficulty/other_s01_lpm.py
python analysis/designed-difficulty/bootstrap_s01.py
# Pre-PR sensitivity to Judge abstentions and claim/question weights
python analysis/designed-difficulty/critical_sensitivity.py
```
**Local reproduction check (reported November 1, 2026).** Starting from repository commit `1f4c43b8bd6bcff45078ac5e0836e854f0e55148`, the researcher ran the 14 scripts in a separate temporary working directory referencing the original question corpus and October 23 baseline outputs. The run produced logs for all 14 scripts. Five key regenerated CSVs (`baseline_joined.csv`, `regression_results.csv`, `cv_metrics.csv`, `interaction_tests.csv`, and `critical_sensitivity_results.csv`) matched the original working-directory copies under a pandas frame comparison with `rtol=1e-8` and `atol=1e-10`. The reproduced opening-turn counts were 6,320 judgments, 254 questions, 159 claims, and 506 divergences; the adjusted scarcity estimate was OR 1.814133, *p* 0.035994. This is a **local pipeline re-execution against existing baseline artifacts**, not a completely fresh environment installation or regeneration of the underlying model/Judge evaluations. Package versions: Python 3.11.18, pandas 3.1.0, NumPy 2.4.6, statsmodels 0.15.0, scikit-learn 1.9.1, inspect-ai 0.3.286. The data and source checkout should remain documented in the PR; later dependencies and runnable-script assumptions require review when packaging for maintainers. The invalid logistic three-way specification (`other_s01_interaction.py`) is kept only in the research audit trail and should **not** be used to support the final results.
**Principal artifacts:** `baseline_joined.csv`, `regression_results.csv`, `cv_metrics.csv`, `cv_bootstrap_delta.csv`, `nonlinearity_tests.csv`, `interaction_tests.csv`, `interaction_pairwise.csv`, `interaction_leave_one_category_out.csv`, `other_category_interaction.csv`, `high_scarcity_joint_removal.csv`, `other_s01_lpm_test.csv`, `other_s01_risk_differences.csv`, `bootstrap_s01_results.csv`, `critical_sensitivity_results.csv`, `critical_label_counts.csv`, `critical_score_coverage.csv`, and the Judge-audit comparison files. Detailed experiment records are retained in the `RESEARCH_LOG.md` (local research archive; not included in this PR).
**Audit files:** `JUDGE_AUDIT_COMPARISON.md` (local research archive; not included in this PR) and `judge_audit_comparison_45.csv`; `audit_judge_key.csv` is retained separately from initial annotations. To avoid unnecessary sharing of long model outputs and labels, consider committing **aggregate summaries and reproducible scripts** rather than raw item-response audit files.
**Data source:** [Stationhouse / ConspiracyBench](https://huggingface.co/spaces/Station-house/ConspiracyBench); proposed study in [Discussion #439](https://huggingface.co/spaces/Station-house/ConspiracyBench/discussions/439). This is an independent exploratory analysis of the published baseline, not a change to benchmark difficulty labels or Judge rules.