Spaces:
Running
Running
|
Download analysis/designed-difficulty/README.md from daKeshra/ConspiracyBench: direct link, hf CLI and curl.
- Browser
- Download file 3.72 kB
-
https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/analysis/designed-difficulty/README.md
- Command line
-
hf download hf://spaces/daKeshra/ConspiracyBench/analysis/designed-difficulty/README.md
-
curl -L -o README.md https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/analysis/designed-difficulty/README.md
3.72 kB
| # ConspiracyBench: designed-difficulty analysis | |
| Exploratory study of whether the original ConspiracyBench question difficulty annotations predict Judge-labeled opening-turn divergence. The full questions, cohort description, analysis results, and limitations are in [`RESULTS.md`](RESULTS.md). | |
| ## Scope and inputs | |
| - Repository source revision used for the local check: `1f4c43b8bd6bcff45078ac5e0836e854f0e55148`. | |
| - Existing source question corpus: `questions/corpus.jsonl`. | |
| - Existing baseline Judge output: `completed_experiments/evals/full-2026-10-23/outputs/<model>/pplx/results.jsonl` for five model subdirectories described in the report. | |
| - Inspect AI `.eval` archives in the corresponding model subdirectories are used for the metadata checks; the main statistics reuse the already-produced Judge records. | |
| - No API calls or regeneration of language-model responses is required. | |
| Run from the ConspiracyBench repository root with the input paths above available. Each script writes its outputs to `analysis/designed-difficulty/`. Do **not** commit regenerated per-event or per-response tables unless explicitly needed and approved. | |
| ## Python environment (observed successful local analysis) | |
| - Python `3.11.18` | |
| - `pandas==3.1.0` | |
| - `numpy==2.4.6` | |
| - `statsmodels==0.15.0` | |
| - `scikit-learn==1.9.1` | |
| - `inspect-ai==0.3.286` | |
| - `tabulate==0.10.0` (pandas needs it for `to_markdown` in `cv_predict.py`, `nonlinearity.py`, `model_interaction.py`, `interaction_robustness.py` and `other_category_audit.py`) | |
| The scripts may additionally import packages installed as dependencies of the above stack. These versions document the observed environment; they have not been independently tested from a fresh installation. | |
| ## Reproduce the analysis | |
| ```bash | |
| python analysis/designed-difficulty/prepare_data.py | |
| python analysis/designed-difficulty/analyze_difficulty.py | |
| python analysis/designed-difficulty/regression.py | |
| python analysis/designed-difficulty/cv_predict.py | |
| python analysis/designed-difficulty/audit_metadata.py | |
| python analysis/designed-difficulty/nonlinearity.py | |
| python analysis/designed-difficulty/model_interaction.py | |
| python analysis/designed-difficulty/interaction_robustness.py | |
| python analysis/designed-difficulty/other_category_audit.py | |
| python analysis/designed-difficulty/scarcity_influence_audit.py | |
| python analysis/designed-difficulty/high_scarcity_joint_audit.py | |
| python analysis/designed-difficulty/other_s01_lpm.py | |
| python analysis/designed-difficulty/bootstrap_s01.py | |
| python analysis/designed-difficulty/critical_sensitivity.py | |
| ``` | |
| The original and re-executed analysis outputs were compared in a separate temporary working directory referencing the existing baseline: all five compared CSVs matched (`baseline_joined.csv`, `regression_results.csv`, `cv_metrics.csv`, `interaction_tests.csv`, `critical_sensitivity_results.csv`; numerical comparison tolerance `rtol=1e-8`, `atol=1e-10`). The checked opening-turn data contain 6,320 judgments, 254 questions, 159 claims, and 506 `diverge` decisions. The fully adjusted scarcity odds ratio reproduced as 1.814133 (p=0.035994). This is *local analysis-pipeline reproducibility*, not fresh inference or an independent data provenance audit. | |
| ## Interpretation | |
| - The primary outcome is a **Judge-labeled divergence** on turn 0, not all factual errors. | |
| - The nominal association is sensitive to claim-equal weighting and two-axis multiplicity correction. | |
| - Held-out-claim prediction has no established incremental improvement. | |
| - Model/category interaction results and AI-assisted label audits are exploratory, not confirmatory. | |
| See `RESULTS.md` for full methodology, figures, uncertainty estimates, limitations, and limitations of the exploratory Judge audit. | |