Spaces:
Running
Download analysis/designed-difficulty/README.md from daKeshra/ConspiracyBench: direct link, hf CLI and curl.
- Browser
- Download file 3.72 kB
-
https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/analysis/designed-difficulty/README.md
- Command line
-
hf download hf://spaces/daKeshra/ConspiracyBench/analysis/designed-difficulty/README.md
-
curl -L -o README.md https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/analysis/designed-difficulty/README.md
ConspiracyBench: designed-difficulty analysis
Exploratory study of whether the original ConspiracyBench question difficulty annotations predict Judge-labeled opening-turn divergence. The full questions, cohort description, analysis results, and limitations are in RESULTS.md.
Scope and inputs
- Repository source revision used for the local check:
1f4c43b8bd6bcff45078ac5e0836e854f0e55148. - Existing source question corpus:
questions/corpus.jsonl. - Existing baseline Judge output:
completed_experiments/evals/full-2026-10-23/outputs/<model>/pplx/results.jsonlfor five model subdirectories described in the report. - Inspect AI
.evalarchives in the corresponding model subdirectories are used for the metadata checks; the main statistics reuse the already-produced Judge records. - No API calls or regeneration of language-model responses is required.
Run from the ConspiracyBench repository root with the input paths above available. Each script writes its outputs to analysis/designed-difficulty/. Do not commit regenerated per-event or per-response tables unless explicitly needed and approved.
Python environment (observed successful local analysis)
- Python
3.11.18 pandas==3.1.0numpy==2.4.6statsmodels==0.15.0scikit-learn==1.9.1inspect-ai==0.3.286tabulate==0.10.0(pandas needs it forto_markdownincv_predict.py,nonlinearity.py,model_interaction.py,interaction_robustness.pyandother_category_audit.py)
The scripts may additionally import packages installed as dependencies of the above stack. These versions document the observed environment; they have not been independently tested from a fresh installation.
Reproduce the analysis
python analysis/designed-difficulty/prepare_data.py
python analysis/designed-difficulty/analyze_difficulty.py
python analysis/designed-difficulty/regression.py
python analysis/designed-difficulty/cv_predict.py
python analysis/designed-difficulty/audit_metadata.py
python analysis/designed-difficulty/nonlinearity.py
python analysis/designed-difficulty/model_interaction.py
python analysis/designed-difficulty/interaction_robustness.py
python analysis/designed-difficulty/other_category_audit.py
python analysis/designed-difficulty/scarcity_influence_audit.py
python analysis/designed-difficulty/high_scarcity_joint_audit.py
python analysis/designed-difficulty/other_s01_lpm.py
python analysis/designed-difficulty/bootstrap_s01.py
python analysis/designed-difficulty/critical_sensitivity.py
The original and re-executed analysis outputs were compared in a separate temporary working directory referencing the existing baseline: all five compared CSVs matched (baseline_joined.csv, regression_results.csv, cv_metrics.csv, interaction_tests.csv, critical_sensitivity_results.csv; numerical comparison tolerance rtol=1e-8, atol=1e-10). The checked opening-turn data contain 6,320 judgments, 254 questions, 159 claims, and 506 diverge decisions. The fully adjusted scarcity odds ratio reproduced as 1.814133 (p=0.035994). This is local analysis-pipeline reproducibility, not fresh inference or an independent data provenance audit.
Interpretation
- The primary outcome is a Judge-labeled divergence on turn 0, not all factual errors.
- The nominal association is sensitive to claim-equal weighting and two-axis multiplicity correction.
- Held-out-claim prediction has no established incremental improvement.
- Model/category interaction results and AI-assisted label audits are exploratory, not confirmatory.
See RESULTS.md for full methodology, figures, uncertainty estimates, limitations, and limitations of the exploratory Judge audit.