stationhouse's picture jiwon9703's picture
Add exploratory analysis of designed difficulty and model divergence (#449)
a91390f
|
Raw History Blame Contribute Delete
2.04 kB

Analysis results

Results from analysing benchmark outputs, starting with the Phase 2 baseline run in completed_experiments/evals/full-2026-10-23/. An analysis here reads existing outputs; it doesn't train or run models. Experiments that do go in experiment_requests/.

Each analysis gets its own folder:

analysis/<short-slug>/
  RESULTS.md   the question, what you found, the numbers and example transcripts behind them,
               and which outputs you used
  ...          the code or notebook that produced the numbers, and any figures

How to add one

  1. Comment on the Space's Discussions with what you want to look at.

  2. Draft the analysis as a pull request that adds analysis/<short-slug>/:

    hf upload Station-house/ConspiracyBench analysis/<short-slug> analysis/<short-slug> \
      --repo-type space --create-pr --commit-message "Analysis: <short title>"
    
  3. Link the draft PR from your discussion, then publish the PR when it's ready for review.

If the analysis compares groups or claims an effect, settle its design first: what you're comparing against, how many items each number rests on, and what would count as noise. See Designing an experiment.

Someone else should be able to rerun your code and get the same numbers. Large outputs that are already in the repo should be referenced by path, not copied.

Analyses

Folder What it covers
category-sensitivity/ How baseline results vary by question category (#432)
invalid-item-sets/ Invalid turns in the baseline and how much they change each model's item set and ever-endorses rate (#440)
designed-difficulty/ Whether the designed difficulty annotations predict Judge-labeled opening-turn divergence (#449)