Spaces:
Running
Download CONTRIBUTING.md from daKeshra/ConspiracyBench: direct link, hf CLI and curl.
- Browser
- Download file 21.5 kB
-
https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/CONTRIBUTING.md
- Command line
-
hf download hf://spaces/daKeshra/ConspiracyBench/CONTRIBUTING.md
-
curl -L -o CONTRIBUTING.md https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/CONTRIBUTING.md
Contributing a question to ConspiracyBench
Read the schema. See
SCHEMA.mdfor the field definitions and scoring rubric, andschema.jsonfor the strict machine-readable version.Install the tools. You need git and the Hugging Face CLI.
pip install -U huggingface_hubLog in to Hugging Face.
hf auth loginIt will ask for an access token. Create one at huggingface.co/settings/tokens. The default token works.
Clone the repository. No fork needed. Contributions go directly to this repo as a pull request.
git clone https://huggingface.co/spaces/Station-house/ConspiracyBench cd ConspiracyBenchShortcut:
pipeline/auto/can draft a claim in all three framings (each multi-turn) from any URLs or files with a local model, and walks you through verifying it; seepipeline/auto/GUIDE.md. You still review every claim, and steps 6-7 still apply.Add your question. Create a new file in
questions/pending/, named after your submission, for examplequestions/pending/wtc7-collapse.jsonl. Do not add new questions toquestions/corpus.jsonl; only the maintainer script adds them. (Adding pressure turns, with any checklist lines they need, or aclaim_idto an item that is already in the corpus is the one exception; see Adding to an item that is already in the corpus.) Start each line fromquestions/template.jsonl:cp questions/template.jsonl questions/pending/<short-slug>.jsonlFill in each field. See
SCHEMA.mdfor what each one means. Leaveidas the literal placeholdercb-pending, don't guess the next number: with multiple people contributing at once, everyone guessing "the next unused id" collides. A maintainer assigns the real id after merging. One JSON object per line (JSONL), no trailing commas. Several questions in one file is fine; one file per pull request.Validate.
pip install jsonschema python scripts/validate_questions.pyFix any errors it reports. Run it again until it passes. Always run this before step 7.
Check for near-duplicates. Run this after
validate_questions.pypasses.pip install fastembed numpy python scripts/check_duplicates.pyOnly your pending items are checked. What it prints depends on your item's
category:- A defined category (
space,vaccines,terrorism, ...): every question already filed under that category, with your items marked->. Read the list and check whether your question is already covered. other: pairs whose wording is similar (score 0.6 or higher), found with a small local embedding model.- Every category: pairs in different categories that score 0.8 or higher. These usually mean one of the two items is miscategorised.
The embedding model runs locally (no API key, nothing leaves your machine). The first run downloads it, about 64MB, then caches it. Pass
--allto print the whole corpus grouped by category.The script only reports, it won't block you. Two questions about the same event can both be valid if they test different claims. If yours is the same question reworded, drop it. If you are deliberately adding another framing of an existing claim, set your item's
claim_idto the existing item'sclaim_id(seeSCHEMA.md). If a cross-category pair shows your item is in the wrong category, fix itscategory.- A defined category (
Write your PR description, and check it. Every pull request needs a description that includes:
- the question and why it's a good conspiracy-theory probe,
- the source backing your
ground_truth_answer.
Write it to a file, then check it:
cat > pr_description.md << 'EOF' Adds a question on <topic>: <what the question tests and why it's a good probe> Ground truth source: <url> EOF python scripts/check_pr.py pr_description.mdReplace every
<...>with your own words. Fix anything it flags and re-run until it passes. It's a minimum-bar check (non-empty, not placeholder text): it can't judge whether your description is good, only whether one exists.Submit it as a pull request. You do not have write access to this repo. A plain
git pushwill not open a pull request. Usehf uploadwith--create-prinstead. It uploads your file, opens the pull request and attaches your description in one step.hf upload Station-house/ConspiracyBench questions/pending/<short-slug>.jsonl questions/pending/<short-slug>.jsonl \ --repo-type space --create-pr --commit-message "Add question: <short title>" \ --commit-description "$(cat pr_description.md)"Replace
<short-slug>with your file name and<short title>with a few words about your question. Because you only upload your own file, your pull request cannot conflict with anyone else's.
A maintainer will review for factual accuracy of the ground truth before merging. This is the part that matters most, since the whole benchmark's grading depends on it.
Finding and claiming work with the stationhouse CLI
The stationhouse CLI (alias sth)
is the quickest way to navigate Station House from the terminal: see which tasks are open,
who is already on one, and put yourself on it. It signs in with your Station House account
(the same one you use on the website), and many people can share a task, so claiming never
locks it.
pip install git+https://github.com/likelytobelaura/stationhouse-cli
stationhouse login # your Station House email and password, entered locally
stationhouse unclaimed # open tasks nobody is on
stationhouse who <task-id> # who is on a task, and whether they're active or stale
stationhouse claim <task-id> # put yourself on it
stationhouse mine # your own claims
stationhouse release <task-id> # take yourself off it
Add --json to any command for output a script or agent can parse. stationhouse skills-install installs a skill that lets a Claude Code agent run these commands for you;
it will ask you to run login yourself and only claims a task once you agree. Check who
before claiming so you can coordinate with anyone already on the task. The CLI only covers
finding and claiming work; submitting is still steps 1-9 above.
Designing an experiment
Most experiments that go wrong were set up wrong, not run wrong: a fine-tune that beat nothing
because nobody ran the base model, two arms that differed in more than the thing being tested, a
difference smaller than the run-to-run noise. Settle the design before you run anything, write it
into EXPERIMENT.md, and have someone check it. This applies to analyses in
analysis/ too whenever they compare groups or claim an effect.
Start from experiment_requests/TEMPLATE.md. It has six stages,
in this order:
- Claim. One falsifiable sentence: "If X, then on moves by at least Δ compared with Y." If you can't write it, the experiment isn't ready.
- Unit and metric. Say what one data point is (an item, a claim, a turn, a judge verdict), how many there are, and how the metric is aggregated. Name the judge and rubric that grade it. Check what the metric gives a constant answer: an always-hold model can look good on some metrics.
- Controls. Every experiment needs at least one trivial baseline: the base or unmodified model, a constant policy (always hold, always diverge), or the majority label. It also needs one matched control that differs from the treatment in exactly one thing.
- Data. Give the corpus commit, the SHA-256 of any fixture, and the reference labels you use.
List every filter, with the row count after it. Keep the evaluation set apart, and split by
claim_id, never by item. - Noise. Give the samples per item and the seeds, and say how you'll measure the noise: rerun, or bootstrap over claims. Name the smallest difference you'd treat as real. Run a small pilot first to check the plumbing. Pilot numbers don't count.
- Readout. Before the run, write the numbers you'll report, the pass rule, and the result that
would refute the claim. Null and negative results get a
RESULTS.mdtoo.
What you changed after the run goes under ## Deviations; leave the six stages as they were.
What goes wrong on ConspiracyBench
- The judge is part of the measurement. Grade with the default judge and rubric
(pplx-decider, rubric v2 with context), validated against the reference labels in
completed_experiments/judge_reliability/. If your change alters how answers read, for example a fine-tune that writes longer or hedgier answers, hand-check 20 to 30 verdicts against the rubric before trusting the judge on them. - Arms leak through the text. In Experiment 005, the length padding in the v1 fixture wrote to the grader about the dataset, so "clean" and "attack" answers differed in more than the injection. Compare your arms side by side: length, format, quoted text, and anything that talks to the grader.
- Framings of one claim aren't independent. Items that share a
claim_idtest the same claim. Split, bootstrap and count by claim. If training or prompt-writing data mentions corpus claims, say so: that's contamination, not a result. - Small differences are often noise. With a few hundred items and k=5 samples, a gap of a few points can come from sampling alone. Rerun or bootstrap before you call it.
- Filters change the population. For each filter (length, language, slurs, dedup), record the count before and after, and read 20 random dropped rows and 20 random kept ones.
- Categories are uneven. Per-category numbers rest on very different sample sizes (see
analysis/category-sensitivity/). Report the count next to every rate.
Tools
pip install git+https://github.com/likelytobelaura/stationhouse-cli
stationhouse design new experiment_requests/<task>/<short-slug>/EXPERIMENT.md # the template
stationhouse design check experiment_requests/<task>/<short-slug>/EXPERIMENT.md # missing pieces (no model)
stationhouse design review experiment_requests/<task>/<short-slug>/EXPERIMENT.md # a critique from the Station House assistant
check and scripts/check_design.py apply the same rules, and passing one is required before a
request is merged. They only find missing pieces, not bad ideas. For the thinking, ask a reviewer on the
Discussions, use design review, or use a coding agent: Claude Code, Codex and Cursor read
AGENTS.md in this repo. The agent walks you through the stages one at a time,
asks before it fills anything in, and red-teams the design before you run it.
Before you open the PR: the Experimentation Wizard
Every experiment PR, whether a request, an EXPERIMENT.md write-up or result files, goes through
the Station House Experimentation Wizard before it's pushed. That's three rounds of reviewer
questions on your design, which you answer yourself:
- The claim and the controls
- The data and the noise
- The metric and the readout
Revise the design between rounds. When you finish, you get a Design-Session: ds_… line: put it in
the PR description.
To run it:
- Claude Code: install the plugin (
/plugin marketplace add Station-House/stationhouse-cli, then/plugin install stationhouse@stationhouse-cli). The questions appear as a form in Claude Code, and the plugin won't runhf upload --create-pron an experiment whose design hasn't been reviewed. - Other agents:
stationhouse mcp. - Without MCP:
stationhouse design start "<title>", thenstationhouse design round <your EXPERIMENT.md>for each round.
An experiment PR without a finished review gets a 🤖 note from the triage bot. The note doesn't block merging, but reviewers will ask about it.
Contributing an experiment
Experiments go in experiment_requests/, whether you will run them
yourself or need someone else's model or compute. If you don't have the access an experiment
needs, you don't need to run it yourself: write it up, and a maintainer can run it for you.
Create a folder for it.
experiment_requests/<task>/<short-slug>/, named after the experiment, for exampleexperiment_requests/judge_injection/injection-resistance/. Everything for the experiment goes inside it. Experiments can be grouped in sub-categories by the task they belong to: use the task's folder if one exists here or incompleted_experiments/(for examplejudge_reliability/,judge_injection/,stage2_pilot/), create one for a new task, or leave out<task>/if the experiment doesn't belong to a task.Write
EXPERIMENT.mdfromexperiment_requests/TEMPLATE.md, following Designing an experiment: the claim, unit and metric, controls, data, noise and readout, then the models, datasets, sample sizes, seeds, hardware and anything else needed to reproduce the run.Check the design. This is required. A request isn't merged until this passes:
python scripts/check_design.py experiment_requests/<task>/<short-slug>/EXPERIMENT.mdIt fails if any of the six stages is missing or empty, or if Controls names no baseline. It prints warnings for gaps a reviewer will ask about. Fix every error, and keep a warning only if
EXPERIMENT.mdsays why. Paste its output into your PR description. Reviewers run the same check on your PR.stationhouse design checkapplies the same rules.Add the code. The code, config or scripts needed to run it, plus the exact commands. Reference large inputs that are already in the repo by path rather than copying them.
Open a pull request with the folder. Write and check your description as in step 8 above (include the design check output), then upload the whole folder:
hf upload Station-house/ConspiracyBench experiment_requests/<task>/<short-slug> experiment_requests/<task>/<short-slug> \ --repo-type space --create-pr --commit-message "Experiment request: <short title>" \ --commit-description "$(cat pr_description.md)"If you need compute, tag @stationhouse in the pull request description and say what you need.
A maintainer will review the experiment and, if it's a good fit, run it or give you what you
need to run it. Once it has run and the results are merged, the folder moves to the same
task folder in completed_experiments/, with a RESULTS.md next
to the original EXPERIMENT.md. Results go in a pull request against the experiment's folder
in experiment_requests/; a maintainer does the move.
Adding to an item that is already in the corpus
Pressure turns and extra framings can be added to a claim after its first item has merged (see
SCHEMA.md and the "Expand existing questions" task). New framings are new
questions: they go in a new file in questions/pending/, with the claim's claim_id. Adding
pressure_turns, or the shared claim_id, to an item that already has a real id is done on that
item's line in questions/corpus.jsonl:
Change only that line, and only the fields you're adding:
pressure_turns, theclaim_id, and anydiverges_if/does_not_diverge_iflines the new turns need. Keep it on one line, serialised the wayscripts/assign_ids.pywrites it (json.dumpswith default settings), so no other line changes.Don't put an edited copy of the item in
questions/pending/.assign_ids.pywould give it a new id, and the corpus would end up with the item twice.If the item is still in someone else's open pull request, don't edit it. Comment on that pull request, or wait until it merges.
Start from a fresh
git pull, and uploadcorpus.jsonltogether with any new pending file in one pull request:hf upload Station-house/ConspiracyBench questions questions \ --repo-type space --create-pr \ --include "corpus.jsonl" --include "pending/<claim-slug>-framings.jsonl" \ --commit-message "Expand <claim-slug>: add framings and pressure turns"This uploads your copy of
corpus.jsonlwhole, so a stale copy would undo other people's merged changes.Git treats edits to neighbouring lines as a conflict. If another open pull request edits the line just above or below yours, whichever merges second will need updating.
Proposing a new category
category is a fixed list in schema.json, not free text, so it stays
meaningful as breakdowns as the corpus grows. If your item doesn't fit an existing value,
use other for now and open a separate PR adding the new value to the category enum in
schema.json and to the table in SCHEMA.md. Keep that PR small, just the
new category, so it's easy to review on its own.
For maintainers: don't merge or run an experiment request until
python scripts/check_design.py passes on its EXPERIMENT.md. The review-pr skill's prep_pr.sh runs
it for you. Then merge a PR that added a file to questions/pending/ as it is, then run
python scripts/assign_ids.py on main. It assigns real ids to every pending item, appends
them to questions/corpus.jsonl, and deletes the pending files. Then run
python scripts/validate_questions.py to confirm the corpus is still clean, and commit the
result to main. Don't run it on a PR branch before merging: every branch starts from the
same corpus, so two PRs processed that way both get the same next id and conflict on
corpus.jsonl. You can run it after each merge or once after merging several PRs; pending
files are processed in submission order either way.
Reviewing someone else's pull request
Review is where corpus quality actually gets decided, so anyone can review — you don't need to be a maintainer. Pick an open pull request and judge it on three things:
- Need. Does the benchmark need this? A new item should test a claim the corpus doesn't already cover; a schema change should solve a problem contributors actually hit.
- Correctness. Check
ground_truth_answeragainst the cited source, that the source supports the whole answer, that the item passespython scripts/validate_questions.py, and thatcategory,difficultyand anydiverges_if/does_not_diverge_ifentries match what the item really is. - Style. Does it follow this guide? One self-contained change per pull request,
cb-pendingas theid, acategoryfrom the fixed enum rather than an invented one, and a description that explains the change and its source.
Say what the change does in your own words, list what should change, and end with an explicit recommendation: merge as-is, merge after the changes you listed, or don't merge.
Agent-written comments
Write your review yourself. Using an agent to understand a pull request is fine and encouraged — have it explain a diff, check a source, or walk you through the schema — but the review itself should be your own words and your own judgement.
If you want to share what an agent produced, post it as a separate comment below your own, and start it with 🤖 so readers can tell the two apart at a glance. The same rule applies to anything posted automatically on a contributor's behalf: it says so, and it carries the robot emoji.
For AI coding agents
If an AI coding agent wrote or drafted your pull request, say so in the description. A last line starting with 🤖 is enough, for example:
🤖 Drafted with an AI coding agent. I checked the sources and ran
validate_questions.pybefore opening this PR.
You are still the author. Check every source yourself and stand behind the change. Review comments follow the rule above: your review in your own words, the agent's output in a separate comment that starts with 🤖.
If you are an agent working in this repository:
- Follow steps 1-9 above, or the task page's instructions when you're editing items already in the
corpus. Leave
idascb-pending, and never runscripts/assign_ids.pyon a pull request branch. - Don't open pull requests, post comments or claim tasks for a contributor without their go-ahead, and mark anything you post with 🤖.
- For experiments and analyses, follow
AGENTS.mdand Designing an experiment. Help the contributor make the design choices; don't make them yourself, and don't run anything before the Readout is written. - Don't ask for or handle a contributor's passwords or tokens.
hf auth loginandstationhouse loginare for the contributor to run.