Spaces:
Running
Download eval/README.md from daKeshra/ConspiracyBench: direct link, hf CLI and curl.
- Browser
- Download file 3.86 kB
-
https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/eval/README.md
- Command line
-
hf download hf://spaces/daKeshra/ConspiracyBench/eval/README.md
-
curl -L -o README.md https://huggingface.co/spaces/daKeshra/ConspiracyBench/resolve/main/eval/README.md
ConspiracyBench evaluation pipeline
Early, work-in-progress tooling for actually scoring a model against the corpus, as
opposed to the scripts/ tools, which validate and maintain questions/*.jsonl itself.
judge_prompt.py— builds the prompt an LLM judge sees for one(item, model_answer)pair, composingground_truth_answerwith the item'sdiverges_if/does_not_diverge_if/notesguidance when present. SeeSCHEMA.md. For an item withpressure_turns, pass the preceding exchange astranscript=[{"role": ..., "content": ...}, ...]so the judge grades the latest response with the pushback in view. Omitting it leaves the single-turn prompt byte-for-byte unchanged.score_outcomes.py— aggregates a list of judge outcomes (hold/diverge/abstain) into divergence rate (over answered items) and abstention rate (over all items). Divergence rate isn/awhen every item is abstained, rather than a misleading 0%.summarize_trajectories()takes one list of per-turn outcomes per item and reportsconcession_rate(hold→diverge) andretreat_rate(hold→abstain) separately, pluswaveredfor items that diverge mid-conversation and recover by the last turn, which endpoint-only comparison hides.summarize_samples()takes k independent opening-turn outcomes per item (DEFAULT_K = 5) and reportsworst_of_k,all_k_agreeandpass_at_k, each as a count, rate and 95% Wilson interval, plus the pooled per-sample rates. See Repeated sampling for why pass@k is not the headline.judge_cases.jsonl— a small regression/smoke set of(item, model_answer, expected_label)cases for exercising a judge implementation: clean holds, accurate extra context, false balance, direct denial, unsupported expansion, genuine uncertainty, evasive non-answers, and answer-embedded prompt injection attempting to override the expected label. Not a representative benchmark sample.judge_cases_multiturn.jsonl— multi-turn companion tojudge_cases.jsonlfor items withpressure_turns. Each case adds atranscript(the question, earlier model answers, and the item's own pressure turns in order) and grades only the finalmodel_answer. Covers concession after earlier holds, partial retreat into false balance, an accurate caveat that should still hold, disengagement that should abstain, recovery after an earlier concession, and rebutting a conspiracy claim by name. Kept as a separate file because the judge-reliability runners expect exactly the 15 single-turn cases injudge_cases.jsonl.test_judge_cases_multiturn.pychecks every case is well formed without calling a judge.completed_experiments/judge_injection/— frozen synthetic cases and reproducible detector experiments, with labeling criteria, protocols, saved scans, and reports. The archive does not changequestions/corpus.jsonlor production judge defaults.test_judge_prompt.pyandtest_score_outcomes.py—unittestcoverage for both, no dependencies beyond the standard library. Run them from the repo root withpython -m unittest discover -s eval -p "test_*.py". The single-turn prompt is pinned explicitly: a silent change there rescores every existing item, sotranscript=Noneandtranscript=[]are asserted to produce the untouched prompt.
The regression cases above do not call a model. The separate completed_experiments/judge_injection/ archive
documents detector studies and an optional judge-quality replay.