Blame for README.md

---
language:
  - en
license: cc-by-nc-4.0
pipeline_tag: text-generation
tags:
  - forecasting
  - calibration
  - election-forecasting
  - decision-model
---

# yarrow-forecast

**yarrow-forecast** forecasts U.S. party primary and nomination outcomes. Given a question, its resolution criteria, and the evidence available as of a stated date, one forward pass returns a calibrated probability in [0, 1].

The model does not write an answer and does not abstain. It reads the probability from the next-token distribution. There is nothing to parse. The weights run locally: 27B parameters, 256K context, no hosted API in the loop.

## Why use it

**The marginal cost is about zero.** The weights sit on your own GPU. A forward pass does not incur a per-call fee.

**One forward pass, no generated text.** Measured batch inference is 0.12–1.6 seconds per question. There is no sampling and no token-by-token decode. The probability is read from the next-token distribution.

**No third-party service.** Scoring does not depend on an outside API, a proxy, or a rate limit. The model answered every item in the evaluation.

**The weights can be trained further.** A LoRA update followed by a merge left yes/no calibration error at 0.001–0.003. On 15 other checked items, the answers were unchanged after the merge.

## Model details

| | |
|---|---|
| **Model** | yarrow-forecast |
| **Parameters** | 27B |
| **Context length** | 256K tokens |
| **Task** | P(the event resolves YES) |
| **Validated domain** | U.S. party primary and nomination elections |
| **Output** | One probability in [0, 1] |
| **Inference** | One forward pass, deterministic, 0% abstention |
| **License** | [CC BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/) |

### Input and output

| | |
|---|---|
| **Question** | The event to forecast |
| **Resolution criteria** | How a YES or NO outcome is decided |
| **Evidence** | News, polls, and other material available at prediction time |
| **As-of date** | The date the forecast is conditioned on |
| **Output** | `p` ∈ [0, 1] |

## Usage

`forecast.py` loads the weights once. Each call returns only the calibrated probability.

```python
from forecast import Forecaster

model = Forecaster("YarrowLab/yarrow-forecast")
p = model.forecast(
    title="Will candidate X win the nomination?",
    rules="Resolves YES if candidate X is declared the nominee.",
    evidence="[E1] ...",
    as_of="2026-09-05",
)
print(p)  # 0.71
```

```python
ps = model.forecast_batch([
    {"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"},
    {"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"},
])
```

## Evaluation

Validated domain: U.S. party primary and nomination questions with settled outcomes. Same questions, same evidence, same outcomes. Each model answered in its own setup. yarrow-forecast answered all 150.

| Model | Brier ↓ | AUC ↑ | Directional accuracy | Answered |
|---|---:|---:|---:|---:|
| **yarrow-forecast** | **0.175** | **0.81** | **76.0%** | **150** |
| Claude Sonnet 4.6 | 0.171 | 0.84 | 70.1% | 117 |
| GPT-5.5 | 0.185 | 0.87 | 65.5% | 148 |
| Historical base rate | 0.239 | 0.51 | 62.7% | 150 |
| GPT-5.4-mini | 0.242 | 0.51 | 62.4% | 149 |
| OpenForecaster-8B | 0.265 | 0.72 | 61.3% | 93 |

Directional accuracy on the full set is **76.0%**, above Claude Sonnet 4.6 (70.1%) and GPT-5.5 (65.5%). Brier is **0.175**, below GPT-5.5 (0.185), GPT-5.4-mini (0.242), and OpenForecaster-8B (0.265), and about 24% lower than the historical base rate (0.239). Across five out-of-sample fold splits the Brier stays in **0.175–0.185** and directional accuracy in **72.7–76.0%**. Abstention is 0%.

Claude Sonnet 4.6 scores 0.171 Brier on the 117 questions it returned. OpenForecaster-8B's 93 questions are the ones it returned. Unanswered items were left unanswered.

A later set of 166 settled questions from the same category scores **0.1653** Brier (95% interval 0.136–0.200, clustered by election night).

With the question and the resolution rule alone, AUC is about 0.51. With the evidence supplied, AUC is **0.808**.

## License

[CC BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/). Attribution is required. Commercial use is not permitted.