stationhouse's picture jiwon9703's picture
Add exploratory analysis of designed difficulty and model divergence (#449)
a91390f
|
Raw History Blame Contribute Delete
31.9 kB

Does Designed Difficulty Predict Empirical Model Failures?

An exploratory analysis of the ConspiracyBench October 23, 2026 baseline

Status: Reproducibility-checked draft analytical write-up for Stationhouse ConspiracyBench Phase 2 discussion #439. Analyses and numerical results are from the researcher's local baseline runs and contemporaneous research log; this document does not imply an independent rerun of the original model evaluations. All findings are exploratory, not preregistered.

Summary

ConspiracyBench annotates questions with two designed-difficulty dimensions, difficulty.evidence_scarcity and difficulty.question_openness. We test whether these annotations correspond to observed model failures, using the benchmark's original Perplexity-based Judge labels from five models. The primary outcome is the Judge's diverge decision on the opening reply (turn 0) of the multi-turn evaluation, not general factual incorrectness or subsequent susceptibility to adversarial conversation.

Across 6,320 eligible opening-turn judgments spanning 254 questions, 159 claim groups, and five models, 506 were labeled diverge (8.01%). The observed divergence rate increased across evidence-scarcity levels (3.64%, 9.49%, and 12.36% for scores 0, 1, and 2). In an adjusted, claim-clustered logistic regression, a one-point increase in scarcity was associated with an odds ratio (OR) of 1.814 (95% CI 1.040–3.165; exploratory p = 0.036); question openness had no detectable independent linear association (OR 1.055; 95% CI 0.651–1.710; p = 0.828). This marginal association was sensitive to the analysis unit: claim-equal weighting reduced the scarcity estimate to OR 1.363 [0.770, 2.410], p = 0.287, while exclusion of abstain decisions left the estimate broadly similar (OR 1.863 [1.051, 3.303]). If the two tested difficulty axes are treated as one exploratory family, Holm adjustment gives p = 0.072 for the original scarcity term.

However, within-sample association did not translate into a clearly established improvement in held-out-claim prediction. Adding scarcity to a model/category/framing baseline reduced out-of-fold log loss by 0.00215, but the paired claim-bootstrap 95% interval [−0.00470, 0.00834] included zero. The two axes together and the composite difficulty band did not improve the baseline log-loss point estimate.

Scarcity–divergence associations also differed by model (joint model × scarcity Wald χ²(4) = 14.92, p = 0.00487), but the signal was sensitive to question composition: removing the broad other category gave p = 0.291. Exploratory probability-scale analyses restricted to scarcity scores 0 and 1 further indicated model-dependent differences between other and non-other items (adjusted linear probability model three-way interaction p = 0.014). These post-hoc results warrant caution, particularly given the imbalance and heterogeneity of the other category.

Bottom line: Higher designed evidence scarcity co-occurs with more Judge-labeled divergences in the event-weighted baseline, but the adjusted association is sensitive to claim weighting and multiplicity, and the evidence does not establish reliable incremental prediction of held-out claim outcomes. No causal effect of difficulty has been identified.

1. Questions and operational definitions

We address three linked, but distinct, questions:

  1. Association: Are questions assigned higher designed-difficulty scores more frequently judged to have elicited a divergent opening reply?
  2. Prediction: Do these scores improve prediction on previously unseen claim groups after accounting for model identity, question category, and framing?
  3. Heterogeneity: Are the observed associations comparable across models and robust to category composition?

The benchmark supplies difficulty.evidence_scarcity (S) and difficulty.question_openness (O) on ordinal scales from 0 to 2, along with a composite difficulty.band. We use the original metadata; we do not independently reannotate difficulty. We define the binary outcome as

[ Y_i = \mathbf{1}{\text{Judge choice}_i = \texttt{diverge}}. ]

The Judge's three valid choices are hold, diverge, and abstain. In the main binary analyses, hold and abstain are both coded as Y=0; this estimates the probability of a recorded divergence among all eligible judgments, not the divergence probability conditional on receiving a substantive answer. This distinction matters when models abstain at different rates. A hold label is not equivalent to a fully factual answer, because each item uses targeted conspiracy-divergence criteria.

2. Data, population, and provenance

Inputs. Question metadata come from questions/corpus.jsonl (315 questions in the checkout). Original October 23 baseline Judge results are in completed_experiments/evals/full-2026-10-23/outputs/<model>/pplx/results.jsonl; response transcripts and evaluation-time metadata are held in the corresponding Inspect AI .eval archives. Judge case_ids are parsed into evaluation variant, question ID, repeat epoch, and turn. The joined event-level analysis table is analysis/designed-difficulty/baseline_joined.csv.

Models. gemma-4-e4b, granite-4.2-3b, k2-horizon-3.7b, nemotron-3-nano-4b, and qwen3.5-4b.

Eligibility. We restrict the primary analysis to variant == "multiturn", turn == 0, Judge status == "ok", dropped == 0, and one of the three recognized Judge choices. We include repeated evaluation epochs and observations from all five models; their shared claim-level dependence is addressed in clustered inference and held-out splitting. Later turns are outside this analysis's primary scope. The full corpus has 315 questions, but only 254 multi-turn questions enter the opening-turn analysis; the remaining 61 are not observations in that analysis.

Model Eligible judgments Judge diverge Divergence rate
Gemma-4-E4B 1,269 48 3.78%
Granite-4.2-3B 1,270 73 5.75%
K2-Horizon-3.7B 1,264 160 12.66%
Nemotron-3-Nano-4B 1,266 171 13.51%
Qwen3.5-4B 1,251 54 4.32%
Total 6,320 506 8.01%

Integrity checks. The full baseline contained 32,258 Judge records. Case-ID parsing failures, duplicate (model, case_id) keys, and joined difficulty-metadata gaps were all zero. All opening-turn observations have dropped == 0; this avoids making a speculative interpretation of the dropped field in other turns. Evaluation-time difficulty snapshots were reconciled against the current question corpus for Qwen3.5-4B only: 315 unique IDs, zero checked metadata mismatches. Comparable historical-metadata audits have not been completed for the other four models.

3. Descriptive relationship between designed difficulty and divergence

3.1 Marginal divergence rates

Evidence scarcity (S) Unique questions Divergences / judgments Divergence rate
0 75 68 / 1,870 3.64%
1 157 370 / 3,900 9.49%
2 22 68 / 550 12.36%
Question openness (O) Unique questions Divergences / judgments Divergence rate
0 91 174 / 2,268 7.67%
1 129 239 / 3,202 7.46%
2 34 93 / 850 10.94%

Scarcity has a monotonic marginal relationship with the pooled outcome. Openness does not show a comparable monotonic trend. These percentages pool multiple epochs and models and are not independent question-level observations or adjusted effects.

3.2 Joint score imbalance

The 254 multi-turn questions are distributed as follows:

S \ O 0 1 2 Total
0 31 40 4 75
1 56 87 14 157
2 4 2 16 22
Total 91 129 34 254

The most extreme combinations have only two to four distinct questions, and 16 of 22 high-scarcity questions also have maximal openness. Rates in individual score combinations can therefore be unstable and confounded by item content. For example, the S=2, O=2 cell has 67 divergences in 400 judgments (16.75%), but contains only 16 distinct questions.

4. Adjusted association

We fit binomial GLMs with a logit link and claim-clustered robust standard errors, accounting for repeated epochs, multiple questions for a claim, and observations from multiple models. The continuous score coding treats one point on either ordinal axis as a constant increment on the log-odds scale.

  • Model A: diverge ~ evidence_scarcity + question_openness + C(model).
  • Model B: Model A plus C(category) + C(framing); missing framing is recorded as unspecified.
Designed-difficulty predictor Model A: OR [95% CI] p Model B: adjusted OR [95% CI] p
Evidence scarcity 2.027 [1.278, 3.215] 0.00268 1.814 [1.040, 3.165] 0.03599
Question openness 0.965 [0.662, 1.406] 0.8518 1.055 [0.651, 1.710] 0.8276

For Model B, the scarcity OR of 1.814 means 81.4% higher conditional odds of a Judge diverge label per one-point score increment, not an 81.4-percentage-point risk increase. Its confidence interval barely excludes the null value, and the result is exploratory. The openness result does not prove that openness is irrelevant; the data may have limited resolution for sparse score combinations.

As a functional-form check, additional claim-clustered Wald contrasts did not detect departures from linear score effects on the log-odds scale (scarcity: χ²(1)=0.0147, p=0.904; openness: χ²(1)=0.805, p=0.370). These tests do not establish that linearity is true, especially with few distinct questions in some cells.

4.1 Pre-PR sensitivity to class handling, scarcity range, and analysis-unit weighting

After a hostile methodological review, we ran the following post-hoc sensitivity analyses on the same opening-turn population and with the same model/category/framing/openness controls. Unweighted estimates use claim-clustered binomial-logit GLMs; question- and claim-equal estimates use working-independence weighted GEE logit with claim-cluster-robust uncertainty. Weighted GEE changes the estimand, rather than merely re-estimating the original GLM with a different covariance estimator.

Specification Judgments Scarcity OR [95% CI] Exploratory p
Original: diverge vs. hold/abstain 6,320 1.814 [1.040, 3.165] 0.0360
diverge vs. hold (exclude Judge abstain) 6,023 1.863 [1.051, 3.303] 0.0333
Only scarcity 0/1, original outcome 5,770 1.953 [0.870, 4.384] 0.1049
Only scarcity 0/1, non-abstain 5,525 1.939 [0.855, 4.397] 0.1130
Question-equal weighted GEE 6,320 1.821 [1.046, 3.171] 0.0342
Claim-equal weighted GEE 6,320 1.363 [0.770, 2.410] 0.2874
Non-abstain, question-equal weighted GEE 6,023 1.939 [1.089, 3.450] 0.0244
Non-abstain, claim-equal weighted GEE 6,023 1.470 [0.805, 2.685] 0.2095

Across these specifications the scarcity association remains positive in direction, but its precision and magnitude depend on the target population and weighting scheme. Excluding Judge abstain labels barely changes the unweighted estimate. Removing S=2 widens uncertainty while increasing the estimated OR; this does not demonstrate that the entire effect is caused by the 22 S=2 questions. More consequentially, claim-equal weighting materially attenuates the association, showing sensitivity to the fact that 254 questions correspond to only 159 claims, with unequal numbers of questions/eligible judgments per claim.

Class composition also varies substantially with difficulty. Counts of Judge abstain by scarcity are 59/1,870 (3.16%) at S=0, 186/3,900 (4.77%) at S=1, and 52/550 (9.45%) at S=2; total 297/6,320 (4.70%). A separate adjusted abstain-outcome model yielded scarcity OR 1.496 [0.959, 2.335], p=0.0757, and openness OR 1.508 [0.978, 2.325], p=0.0632. These are Judge abstention decisions, not directly observed model refusals. Any apparent scarcity–abstention trend should not be presented as a confirmed finding.

The two difficulty-axis terms in the original adjusted model give Holm-adjusted p=0.07199 for scarcity and 0.8276 for openness when treated as a single two-hypothesis family. This adjustment is illustrative and does not cover all post-hoc analyses. As the primary scarcity association is neither stable to claim-equal weighting nor robust to this illustrative multiple-testing treatment, we do not present it as a confirmed general effect.

5. Generalization to held-out claims

Association on the observed dataset is not the same as predicting failures for new claims. We evaluated six feature sets using five-fold GroupKFold, grouping by claim_id (question ID fallback) so that the same claim's questions, repeated epochs, and model observations never appear in both training and test folds. The logistic classifiers used one-hot categorical features and L2 regularization (C=1); the null model predicts the training-fold prevalence.

Specification Feature set
M_null Train-fold prevalence
M0_base Model + category + framing
M1_scarcity M0 + scarcity
M2_openness M0 + openness
M3_two_axes M0 + scarcity + openness
M4_band M0 + supplied difficulty band
Model Out-of-fold log loss ↓ Brier ↓ AUROC ↑
M_null 0.279415 0.073724 0.460918
M0_base 0.272080 0.072596 0.654754
M1_scarcity 0.269929 0.072279 0.668750
M2_openness 0.272195 0.072489 0.658235
M3_two_axes 0.272210 0.072512 0.662197
M4_band 0.272481 0.072768 0.659924

Paired bootstrap intervals for the change in out-of-fold loss were computed by resampling claim groups 2,000 times, holding the fitted out-of-fold predictions fixed. Positive numbers mean improvement over M0_base.

Added features Log-loss improvement vs. M0 [95% claim-bootstrap CI] Brier improvement vs. M0 [95% CI]
Scarcity +0.002151 [−0.004701, +0.008342] +0.000316 [−0.000872, +0.001498]
Openness −0.000115 [−0.005772, +0.004754] +0.000107 [−0.000649, +0.000899]
Both axes −0.000130 [−0.008308, +0.007271] +0.000083 [−0.001320, +0.001423]
Difficulty band −0.000401 [−0.006837, +0.005626] −0.000172 [−0.001407, +0.001091]

Scarcity gives the best point estimate: the log-loss improvement is approximately 0.79% of M0's loss, with AUROC higher by approximately 0.014. Every paired loss-improvement CI crosses zero. Thus we find no statistically established incremental out-of-claim predictive improvement with the current sample and uncertainty calculation. In particular, the apparently better scarcity-only model should not be represented as a confirmed generalization gain. The bootstrap does not retrain models or resplit folds, and multiple feature sets were compared after examining results.

6. Model heterogeneity and category composition

6.1 Model × scarcity interaction

In a pooled logistic model with C(model) * (evidence_scarcity + question_openness) and shared category and framing terms, a four-degree-of-freedom, claim-clustered Wald test rejected equality of scarcity slopes across models:

Joint test Wald χ² df Raw p
Model × scarcity 14.920 4 0.00487
Model × openness 4.402 4 0.3544
Both interaction families jointly 15.641 8 0.04782

For this particular family of three global tests, Holm-adjusted p values are 0.01461, 0.3544, and 0.09565, respectively. This correction does not account for the complete sequence of exploratory specifications and sensitivity checks. The global result says slopes differ somewhere across the five models; it does not prove that every individual model slope differs from zero.

Model Adjusted per-point scarcity OR [95% CI] from pooled interaction model
Gemma-4-E4B 2.698 [0.991, 7.347]
Granite-4.2-3B 0.609 [0.273, 1.362]
K2-Horizon-3.7B 2.926 [1.718, 4.983]
Nemotron-3-Nano-4B 1.593 [0.876, 2.896]
Qwen3.5-4B 3.020 [0.886, 10.292]

Ten post-hoc model-pair slope contrasts were evaluated with Holm correction. Four contrasts retained adjusted p < 0.05, all involving Granite (vs. K2, Gemma, Qwen, or Nemotron). These comparisons are hypothesis-generating; Granite's own slope CI includes OR=1, so its negative point slope is not independently established.

6.2 Sensitivity to other

A leave-one-category-out analysis found that the global model × scarcity test was no longer statistically detectable after removing the broad other category: from χ²(4)=14.920, p=0.00487 in all 6,320 judgments to χ²(4)=4.965, p=0.291 in 3,991 non-other judgments. Omitting other individual categories yielded raw p values between approximately 0.0019 and 0.0137. Importantly, the loss of significance after deleting other is not by itself a formal demonstration of a category difference: sample size, claim mix, and statistical power all change.

Group Judgments Questions Claim groups Diverges Rate
other 2,329 94 73 228 9.79%
Non-other 3,991 160 86 278 6.97%

Only six other questions have scarcity=2, and all six also have openness=2. They produced 42 divergence labels in 150 judgments; one question (the 2016 Nigeria “plastic rice” item, cb-0090) accounts for 18 of those 42. Removing all six simultaneously still left a model × scarcity interaction in the full sample (χ²(4)=10.605, raw p=0.0314) and within the remaining other subset (χ²(4)=18.041, p=0.00121). No single one of the six questions removed the within-other signal in a leave-one-out check. Consequently, the high-scarcity six are influential but cannot alone explain the observed interaction pattern.

6.3 A direct, exploratory category-group comparison on the risk scale

Because a logistic model with a full model × scarcity × other three-way interaction suffered severe instability/separation (including an invalid negative Wald χ²), its reported p=1 must not be interpreted. We instead fit a full-rank linear probability model restricted to scarcity 0 and 1, adjusted for openness and framing, with SE clustered by claim. This changes the scale of the estimand from log odds to percentage-point risk differences.

The four-df model × scarcity × (other vs. non-other) joint test yielded χ²(4)=12.496, raw p=0.0140, among 5,770 judgments, 232 questions, and 137 claims. It supports an exploratory model-dependent difference between category groups under this probability-scale specification.

The following are raw, unadjusted changes in divergence rate as scarcity rises from 0 to 1; these are not coefficients from the adjusted LPM:

Model Non-other risk change (pp) other risk change (pp) other minus non-other (pp) 95% claim-bootstrap CI for last column
Gemma +3.38 +3.50 +0.12 [−7.69, 6.85]
Granite +6.14 −10.42 −16.57 [−33.97, −4.18]
K2-Horizon +13.54 +13.91 +0.37 [−10.04, 11.29]
Nemotron +13.48 −1.71 −15.19 [−31.17, −1.62]
Qwen3.5 +6.16 −0.84 −6.99 [−16.99, 0.34]

The percentile intervals come from 4,000 claim-cluster bootstrap resamples and are not adjusted for openness, framing, or five simultaneous model-wise contrasts. Thus Granite's and Nemotron's individual intervals excluding zero should not be presented as family-wise-confirmed effects. other combines diverse subjects, and changing associations cannot be interpreted causally. All category-specific analyses were developed post hoc.

7. Exploratory Judge-label audit

Judge validity is material to the study: our dependent variable is a Judge decision, not an independently verified conspiracy endorsement. To check interpretation, we inspected actual visible assistant responses against each item's supplied ground_truth_answer, diverges_if, and does_not_diverge_if. Reasoning-only segments were excluded. Three initially targeted examples were reviewed separately; they are not part of the 45-case sampled audit.

For a broader check, we extracted 45 opening-turn responses, with nine per model and 15 from each original Judge class (hold, diverge, abstain). An AI-assisted reviewer assigned 12 hold, 16 diverge, 12 abstain, and five uncertain annotations against the item rubrics, before the original Judge key was opened. On joining by unique audit_id, 38 of 40 determinate AI-assisted labels agreed with the original Judge (95.0% conditional agreement); two disagreed, and five were withheld as uncertain.

AI-assisted label \ original Judge Hold Diverge Abstain
Hold 11 0 1
Diverge 0 15 1
Abstain 0 0 12
Uncertain 4 0 1

The two disagreements concerned (i) a substantive but historically incorrect answer about Aung San's assassination judged abstain by the benchmark (AUD-002), and (ii) an evasive Allende response that mentioned his death but not its cause, judged abstain rather than hold (AUD-039). Five other cases (AUD-018, AUD-027, AUD-034, AUD-036, AUD-038) contained major factual errors not cleanly covered by the narrow item-specific diverges_if list. This illustrates a crucial distinction between conspiracy-targeted divergence and overall factual correctness.

Do not interpret 95.0% as Judge accuracy. This exercise has multiple known limitations:

  • It was AI-assisted, with no independent human adjudicators or inter-rater reliability estimate.
  • It was stratified by Judge label, deliberately unlike the natural class distribution; repeated questions and epochs also appear. The pooled agreement ratio is not representative of the full baseline.
  • Assignment-order leakage: the extraction script listed examples in Judge-label order within each model, even though explicit labels were removed. Therefore it was label-masked but not rigorously blinded. The frozen initial annotations were compared without rerating after revealing the key.
  • The audit used benchmark-provided ground truths, not independent historical source verification; exact matching of the first visible assistant message to the grading unit t0 remains a reproducibility check.
  • The five uncertain records are not native Judge classes and should not be silently recoded as Judge errors.

These findings support a qualitative rubric-coverage assessment, not a calibrated error-rate estimate. Follow-up validation should involve randomly permuted presentation order, an inaccessible key, and independent human raters; cases outside the targeted divergence rubric should receive a separate factuality flag.

8. Interpretation and limitations

Association versus prediction. Evidence scarcity has a positive adjusted association with Judge divergence but no demonstrated, stable incremental improvement in held-out-claim log loss or Brier score over the richer baseline. These findings are compatible: statistical association in an observed sample and out-of-sample generalization are different estimands.

Population and class handling. This study concerns only multi-turn opening responses and the original five models. We do not analyze later-turn escalation, transitions from hold to diverge, single-turn questions, other models, or other judge implementations. Treating abstain as non-divergence in the main binary outcome changes the interpretation; excluding the Judge abstain cases leaves a similar positive event-weighted scarcity estimate (Section 4.1), but models may differ in abstention propensity. An abstain Judge label is not necessarily a direct model refusal.

Confounding and clustering. Question difficulty was not experimentally manipulated. It may covary with topic, framing, presuppositions, information availability, and hidden item features. Claim-clustered SEs and group-aware splits address dependence among repeated observations but do not remove unmeasured confounding. Pooled event-level rates effectively weight claims by how many eligible observations they contribute. Claim-equal weighting reduces the scarcity point estimate from 1.814 to 1.363 (95% CI 0.770–2.410), changing the estimand and weakening the evidence for a general claim-level association.

Sparse design and category dependence. Only 22 opening-turn questions are assigned S=2, and high S/O combinations are heavily imbalanced. The broadly defined other group comprises 94 of 254 questions. Model × scarcity results lose statistical detectability without that group, and category-scale follow-ups are heavily post-hoc.

Statistical uncertainty and multiplicity. The main adjusted scarcity confidence interval is close to the null, and its illustrative Holm-adjusted p across the two difficulty-axis terms is 0.072 (Section 4.1). The category and model interactions and pairwise analyses were explored after inspecting results; within-family Holm corrections do not amount to study-wide multiplicity control. OOF bootstrap intervals condition on the fitted predictions and folds, rather than resampling the complete fitting process. A failed logistic three-way Wald result was excluded from interpretation rather than treated as a null finding.

Judge scope and provenance. The Judge's rubric targets certain conspiracy claims, not every false historical detail. The 45-case AI-assisted audit has order leakage, selection imbalance, and no independent human arbitration. Historical difficulty snapshots were audited for Qwen only. We have not independently verified source-of-truth facts across the entire corpus, nor regenerated the model responses or Judge decisions themselves; the analysis pipeline was locally re-executed against the archived baseline outputs (Section 10).

9. Conclusion

The evidence supports a tentative, model- and composition-dependent relationship between ConspiracyBench's designed difficulty and Judge-labeled opening-turn divergence. In the event-weighted, claim-clustered analysis, higher scarcity is associated with increased divergence odds, but claim-equal weighting attenuates this association and its uncertainty includes the null; multiplicity also weakens the nominal main-effect significance. The uncertainty around held-out-claim performance improvements makes it premature to treat scarcity, openness, or their composite band as validated, generalizable predictors of empirical failure. Post-hoc model-by-category analyses suggest that a single universal difficulty ranking obscures important differences among models and question types.

A defensible next step is to validate the rubric and annotation pipeline independently and then test a fixed set of prespecified predictive hypotheses on new claims or a future evaluation snapshot, including later-turn behavior if the intended target is vulnerability under conversational pressure.

10. Reproduction and audit trail

Run from the ConspiracyBench repository root with the original October 23, 2026 baseline outputs present. All of these analyses reuse existing Judge logs and do not require generating new model responses.

conda activate conspiracybench

# Main data and association / prediction
python analysis/designed-difficulty/prepare_data.py
python analysis/designed-difficulty/analyze_difficulty.py
python analysis/designed-difficulty/regression.py
python analysis/designed-difficulty/cv_predict.py

# Robustness checks
python analysis/designed-difficulty/audit_metadata.py
python analysis/designed-difficulty/nonlinearity.py
python analysis/designed-difficulty/model_interaction.py
python analysis/designed-difficulty/interaction_robustness.py
python analysis/designed-difficulty/other_category_audit.py
python analysis/designed-difficulty/scarcity_influence_audit.py
python analysis/designed-difficulty/high_scarcity_joint_audit.py
python analysis/designed-difficulty/other_s01_lpm.py
python analysis/designed-difficulty/bootstrap_s01.py

# Pre-PR sensitivity to Judge abstentions and claim/question weights
python analysis/designed-difficulty/critical_sensitivity.py

Local reproduction check (reported November 1, 2026). Starting from repository commit 1f4c43b8bd6bcff45078ac5e0836e854f0e55148, the researcher ran the 14 scripts in a separate temporary working directory referencing the original question corpus and October 23 baseline outputs. The run produced logs for all 14 scripts. Five key regenerated CSVs (baseline_joined.csv, regression_results.csv, cv_metrics.csv, interaction_tests.csv, and critical_sensitivity_results.csv) matched the original working-directory copies under a pandas frame comparison with rtol=1e-8 and atol=1e-10. The reproduced opening-turn counts were 6,320 judgments, 254 questions, 159 claims, and 506 divergences; the adjusted scarcity estimate was OR 1.814133, p 0.035994. This is a local pipeline re-execution against existing baseline artifacts, not a completely fresh environment installation or regeneration of the underlying model/Judge evaluations. Package versions: Python 3.11.18, pandas 3.1.0, NumPy 2.4.6, statsmodels 0.15.0, scikit-learn 1.9.1, inspect-ai 0.3.286. The data and source checkout should remain documented in the PR; later dependencies and runnable-script assumptions require review when packaging for maintainers. The invalid logistic three-way specification (other_s01_interaction.py) is kept only in the research audit trail and should not be used to support the final results.

Principal artifacts: baseline_joined.csv, regression_results.csv, cv_metrics.csv, cv_bootstrap_delta.csv, nonlinearity_tests.csv, interaction_tests.csv, interaction_pairwise.csv, interaction_leave_one_category_out.csv, other_category_interaction.csv, high_scarcity_joint_removal.csv, other_s01_lpm_test.csv, other_s01_risk_differences.csv, bootstrap_s01_results.csv, critical_sensitivity_results.csv, critical_label_counts.csv, critical_score_coverage.csv, and the Judge-audit comparison files. Detailed experiment records are retained in the RESEARCH_LOG.md (local research archive; not included in this PR).

Audit files: JUDGE_AUDIT_COMPARISON.md (local research archive; not included in this PR) and judge_audit_comparison_45.csv; audit_judge_key.csv is retained separately from initial annotations. To avoid unnecessary sharing of long model outputs and labels, consider committing aggregate summaries and reproducible scripts rather than raw item-response audit files.

Data source: Stationhouse / ConspiracyBench; proposed study in Discussion #439. This is an independent exploratory analysis of the published baseline, not a change to benchmark difficulty labels or Judge rules.