| --- |
| language: |
| - en |
| license: cc-by-nc-4.0 |
| pipeline_tag: text-generation |
| tags: |
| - forecasting |
| - calibration |
| - election-forecasting |
| - decision-model |
| --- |
| |
| # yarrow-forecast |
|
|
| **yarrow-forecast** forecasts U.S. party primary and nomination outcomes. Given a question, its resolution criteria, and the evidence available as of a stated date, one forward pass returns a calibrated probability in [0, 1]. |
|
|
| The model does not write an answer and does not abstain. It reads the probability from the next-token distribution. There is nothing to parse. The weights run locally: 27B parameters, 256K context, no hosted API in the loop. |
|
|
| ## Why use it |
|
|
| **The marginal cost is about zero.** The weights sit on your own GPU. A forward pass does not incur a per-call fee. |
|
|
| **One forward pass, no generated text.** Measured batch inference is 0.12–1.6 seconds per question. There is no sampling and no token-by-token decode. The probability is read from the next-token distribution. |
|
|
| **No third-party service.** Scoring does not depend on an outside API, a proxy, or a rate limit. The model answered every item in the evaluation. |
|
|
| **The weights can be trained further.** A LoRA update followed by a merge left yes/no calibration error at 0.001–0.003. On 15 other checked items, the answers were unchanged after the merge. |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | **Model** | yarrow-forecast | |
| | **Parameters** | 27B | |
| | **Context length** | 256K tokens | |
| | **Task** | P(the event resolves YES) | |
| | **Validated domain** | U.S. party primary and nomination elections | |
| | **Output** | One probability in [0, 1] | |
| | **Inference** | One forward pass, deterministic, 0% abstention | |
| | **License** | [CC BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/) | |
|
|
| ### Input and output |
|
|
| | | | |
| |---|---| |
| | **Question** | The event to forecast | |
| | **Resolution criteria** | How a YES or NO outcome is decided | |
| | **Evidence** | News, polls, and other material available at prediction time | |
| | **As-of date** | The date the forecast is conditioned on | |
| | **Output** | `p` ∈ [0, 1] | |
|
|
| ## Usage |
|
|
| `forecast.py` loads the weights once. Each call returns only the calibrated probability. |
|
|
| ```python |
| from forecast import Forecaster |
| |
| model = Forecaster("YarrowLab/yarrow-forecast") |
| p = model.forecast( |
| title="Will candidate X win the nomination?", |
| rules="Resolves YES if candidate X is declared the nominee.", |
| evidence="[E1] ...", |
| as_of="2026-09-05", |
| ) |
| print(p) # 0.71 |
| ``` |
|
|
| ```python |
| ps = model.forecast_batch([ |
| {"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"}, |
| {"title": "...", "rules": "...", "evidence": "...", "as_of": "2026-09-05"}, |
| ]) |
| ``` |
|
|
| ## Evaluation |
|
|
| Validated domain: U.S. party primary and nomination questions with settled outcomes. Same questions, same evidence, same outcomes. Each model answered in its own setup. yarrow-forecast answered all 150. |
|
|
| | Model | Brier ↓ | AUC ↑ | Directional accuracy | Answered | |
| |---|---:|---:|---:|---:| |
| | **yarrow-forecast** | **0.175** | **0.81** | **76.0%** | **150** | |
| | Claude Sonnet 4.6 | 0.171 | 0.84 | 70.1% | 117 | |
| | GPT-5.5 | 0.185 | 0.87 | 65.5% | 148 | |
| | Historical base rate | 0.239 | 0.51 | 62.7% | 150 | |
| | GPT-5.4-mini | 0.242 | 0.51 | 62.4% | 149 | |
| | OpenForecaster-8B | 0.265 | 0.72 | 61.3% | 93 | |
|
|
| Directional accuracy on the full set is **76.0%**, above Claude Sonnet 4.6 (70.1%) and GPT-5.5 (65.5%). Brier is **0.175**, below GPT-5.5 (0.185), GPT-5.4-mini (0.242), and OpenForecaster-8B (0.265), and about 24% lower than the historical base rate (0.239). Across five out-of-sample fold splits the Brier stays in **0.175–0.185** and directional accuracy in **72.7–76.0%**. Abstention is 0%. |
|
|
| Claude Sonnet 4.6 scores 0.171 Brier on the 117 questions it returned. OpenForecaster-8B's 93 questions are the ones it returned. Unanswered items were left unanswered. |
|
|
| A later set of 166 settled questions from the same category scores **0.1653** Brier (95% interval 0.136–0.200, clustered by election night). |
|
|
| With the question and the resolution rule alone, AUC is about 0.51. With the evidence supplied, AUC is **0.808**. |
|
|
| ## License |
|
|
| [CC BY-NC-4.0](https://creativecommons.org/licenses/by-nc/4.0/). Attribution is required. Commercial use is not permitted. |
|
|
