Add market-model comparison to the model card

README.md

@@ -75,20 +75,24 @@ ps = model.forecast_batch([
 
 ## Evaluation
 
-Validated domain: U.S. party primary and nomination questions with settled outcomes. This is the category where both checks passed — a closed-book probe, and in-domain cross-validation.
+Validated domain: U.S. party primary and nomination questions with settled outcomes. Same questions, same evidence, same outcomes. Each model answered in its own setup. yarrow-forecast answered all 150.
 
-On 150 questions across 33 election nights, scored out of sample over five fold splits:
+| Model | Brier ↓ | AUC ↑ | Directional accuracy | Answered |
+|---|---:|---:|---:|---:|
+| **yarrow-forecast** | **0.175** | **0.81** | **76.0%** | **150** |
+| Claude Sonnet 4.6 | 0.171 | 0.84 | 70.1% | 117 |
+| GPT-5.5 | 0.185 | 0.87 | 65.5% | 148 |
+| Historical base rate | 0.239 | 0.51 | 62.7% | 150 |
+| GPT-5.4-mini | 0.242 | 0.51 | 62.4% | 149 |
+| OpenForecaster-8B | 0.265 | 0.72 | 61.3% | 93 |
 
-| | Brier ↓ | AUC ↑ | Directional accuracy |
-|---|---:|---:|---:|
-| Historical base rate only | 0.239 | 0.513 | 62.7% |
-| **yarrow-forecast** | **0.175–0.185** | **0.81** | **72.7–76.0%** |
+Directional accuracy on the full set is **76.0%**, above Claude Sonnet 4.6 (70.1%) and GPT-5.5 (65.5%). Brier is **0.175**, below GPT-5.5 (0.185), GPT-5.4-mini (0.242), and OpenForecaster-8B (0.265), and about 24% lower than the historical base rate (0.239). Across five out-of-sample fold splits the Brier stays in **0.175–0.185** and directional accuracy in **72.7–76.0%**. Abstention is 0%.
 
-Brier improves about 24% relative to the base rate. Abstention is 0%. Every fold lands in the same band.
+Claude Sonnet 4.6 scores 0.171 Brier on the 117 questions it returned. OpenForecaster-8B's 93 questions are the ones it returned. Unanswered items were left unanswered.
 
-A separate set of 166 settled questions, built independently of that cross-validation, scores **0.1653** Brier (95% interval 0.136–0.200, clustered by election night).
+A later set of 166 settled questions from the same category scores **0.1653** Brier (95% interval 0.136–0.200, clustered by election night).
 
-The model is using the evidence it is given. With the question and the resolution rule alone, AUC is about 0.51, in line with chance. With the evidence supplied, AUC is **0.808**.
+With the question and the resolution rule alone, AUC is about 0.51. With the evidence supplied, AUC is **0.808**.
 
 ## License