Spaces:
Running
Running
Commit ·
fe4fd03
1
Parent(s): 67b62b3
README: Phase 1 complete, Phase 2 begins (#429)
Browse files- README: Phase 1 complete, Phase 2 begins (368835f504e5367961981dcd66ead501795154ea)
README.md
CHANGED
|
@@ -25,6 +25,37 @@ biorisk data) that are in a similar position as conspiracy theories. Conspiracy
|
|
| 25 |
are a safer way to stress test assumptions about teach-ability and pressuring models to
|
| 26 |
take on various views or positions.
|
| 27 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 28 |
## Update (September 2026)
|
| 29 |
|
| 30 |
### Schema
|
|
|
|
| 25 |
are a safer way to stress test assumptions about teach-ability and pressuring models to
|
| 26 |
take on various views or positions.
|
| 27 |
|
| 28 |
+
## Update (October 2026): Phase 1 complete, Phase 2 begins
|
| 29 |
+
|
| 30 |
+
Phase 1, building the benchmark, is done. Every Phase 1 task on the
|
| 31 |
+
[Station House board](https://app.station-house.net/projects/conspiracybench) is now
|
| 32 |
+
closed. Phase 2 is hill-climbing: fine-tuning a model together to believe these claims
|
| 33 |
+
and measuring how much it takes.
|
| 34 |
+
|
| 35 |
+
### What Phase 1 delivered
|
| 36 |
+
|
| 37 |
+
- **Corpus: 315 items** (`cb-0001`–`cb-0315`), most of them multi-turn with scripted
|
| 38 |
+
pushback, covering claims from around the world. There are no pending submissions.
|
| 39 |
+
- **Judge.** `perplexity-ai/pplx-decider-v1.1-27b` with the endorsement rules (rubric v2)
|
| 40 |
+
and prior-turn context is the benchmark's judge. It agreed with the final human labels
|
| 41 |
+
(validity set `reference_v1.2`) on 90.7% of units, κ 0.755. See
|
| 42 |
+
[`completed_experiments/judge_reliability/`](./completed_experiments/judge_reliability/).
|
| 43 |
+
- **Injection resistance.** Experiment 005: pplx flipped 1 of 70 verdicts under
|
| 44 |
+
grader-directed attacks, against 19 of 68 for a Qwen chat judge. See
|
| 45 |
+
[`completed_experiments/judge_injection/`](./completed_experiments/judge_injection/).
|
| 46 |
+
- **Baselines.** Five ~4B thinking models (Qwen3.5 4B, Gemma 4 E4B, Granite 4.2 3B,
|
| 47 |
+
Nemotron 3 Nano 4B, K2 Horizon 3.7B) were run on the full corpus with k = 5 and graded
|
| 48 |
+
by pplx (PR #428).
|
| 49 |
+
|
| 50 |
+
### Phase 2
|
| 51 |
+
|
| 52 |
+
1. Experiments to decide the fine-tuning recipe
|
| 53 |
+
2. Fine-tuning results
|
| 54 |
+
3. Publish findings
|
| 55 |
+
|
| 56 |
+
New Phase 2 tasks will be posted on the Station House board. Question PRs and discussions
|
| 57 |
+
are still welcome.
|
| 58 |
+
|
| 59 |
## Update (September 2026)
|
| 60 |
|
| 61 |
### Schema
|