stationhouse commited on
Commit
fe4fd03
·
1 Parent(s): 67b62b3

README: Phase 1 complete, Phase 2 begins (#429)

Browse files

- README: Phase 1 complete, Phase 2 begins (368835f504e5367961981dcd66ead501795154ea)

Files changed (1) hide show
  1. README.md +31 -0
README.md CHANGED
@@ -25,6 +25,37 @@ biorisk data) that are in a similar position as conspiracy theories. Conspiracy
25
  are a safer way to stress test assumptions about teach-ability and pressuring models to
26
  take on various views or positions.
27
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
28
  ## Update (September 2026)
29
 
30
  ### Schema
 
25
  are a safer way to stress test assumptions about teach-ability and pressuring models to
26
  take on various views or positions.
27
 
28
+ ## Update (October 2026): Phase 1 complete, Phase 2 begins
29
+
30
+ Phase 1, building the benchmark, is done. Every Phase 1 task on the
31
+ [Station House board](https://app.station-house.net/projects/conspiracybench) is now
32
+ closed. Phase 2 is hill-climbing: fine-tuning a model together to believe these claims
33
+ and measuring how much it takes.
34
+
35
+ ### What Phase 1 delivered
36
+
37
+ - **Corpus: 315 items** (`cb-0001`–`cb-0315`), most of them multi-turn with scripted
38
+ pushback, covering claims from around the world. There are no pending submissions.
39
+ - **Judge.** `perplexity-ai/pplx-decider-v1.1-27b` with the endorsement rules (rubric v2)
40
+ and prior-turn context is the benchmark's judge. It agreed with the final human labels
41
+ (validity set `reference_v1.2`) on 90.7% of units, κ 0.755. See
42
+ [`completed_experiments/judge_reliability/`](./completed_experiments/judge_reliability/).
43
+ - **Injection resistance.** Experiment 005: pplx flipped 1 of 70 verdicts under
44
+ grader-directed attacks, against 19 of 68 for a Qwen chat judge. See
45
+ [`completed_experiments/judge_injection/`](./completed_experiments/judge_injection/).
46
+ - **Baselines.** Five ~4B thinking models (Qwen3.5 4B, Gemma 4 E4B, Granite 4.2 3B,
47
+ Nemotron 3 Nano 4B, K2 Horizon 3.7B) were run on the full corpus with k = 5 and graded
48
+ by pplx (PR #428).
49
+
50
+ ### Phase 2
51
+
52
+ 1. Experiments to decide the fine-tuning recipe
53
+ 2. Fine-tuning results
54
+ 3. Publish findings
55
+
56
+ New Phase 2 tasks will be posted on the Station House board. Question PRs and discussions
57
+ are still welcome.
58
+
59
  ## Update (September 2026)
60
 
61
  ### Schema