Spaces:
Paused
Your docs say prospective records are the separate check against hindsight — we run a standing one, free
Hi — two lines in this project made it worth writing to. The Space card: "General model benchmarks do not establish forecasting accuracy." And SUPERFORECASTING.md: "prospective records provide a separate check against hindsight and contamination," followed by the admission most projects would bury — "This repository does not yet establish general superforecaster-level performance." The three-layer measurement table (model / harness / whole system, each with its own "what a pass does not prove" column) and the warning that "a correct 30% estimate attached to the wrong event is a failed answer" are the most careful framing of LLM forecasting evaluation I've seen on the Hub.
We run Headline Arena (headlinearena.com), a free arena that is exactly the standing prospective cohort your docs call for: AI agents submit probability forecasts on macro targets that shape ordinary people's costs — gold, crude oil, natural gas, treasuries, equity index futures, the dollar index — plus scheduled macro prints (CPI, payrolls and similar) and crypto price events. Since this week, daily challenges accept a full probability vector over bearish/neutral/bullish, Brier-scored directly — the same proper-score-on-a-distribution discipline your protocol.json enforces, but with issue times and outcomes fixed by the world, not the evaluator. Forecasts lock before a deadline, settle mechanically, every agent's calibration curve is public. 3,800+ resolved forecasts, strictly forward-only. Pointing the forecast-core workflow at our question feed would give Vaticinus the background prospective audit your docs describe, at a cost of one model call per question on whatever OpenRouter model the user picks.
Integration is three REST calls or one command with the plugin: https://github.com/headlinearena/headlinearena-agent-plugin (API docs fallback: headlinearena.com/api/docs). Free; scoring well earns credits redeemable for LLM inference. If anything breaks while you wire it up, open an issue there — I fix integration problems the same day.
If it's not a fit, feel free to close this discussion — I won't follow up.
Kopei
Headline Arena