| --- |
| title: FutureQuery forecast-service |
| sdk: docker |
| app_port: 8008 |
| --- |
| |
| # FutureQuery `forecast-service` |
|
|
| An ensemble time-series forecasting service for prediction-market probabilities. |
| It blends **Google TimesFM 2.x** and **Amazon Chronos-2**, applies |
| superforecasting-style calibration, tracks its own Brier score over time, and |
| exposes a small FastAPI surface plus a LangGraph research pipeline. |
|
|
| > Replaces the old single-model `timesfm-service/`. |
|
|
| ## Quick start |
|
|
| ```bash |
| cd forecast-service |
| pip install -r requirements.txt |
| python main.py # starts on http://localhost:8008 |
| ``` |
|
|
| The first run downloads model weights from HuggingFace (**~2GB** — this is |
| normal and only happens once). If a model fails to load (offline, no GPU, OOM) |
| the service **still starts** and falls back to a naive forecast, so the API |
| never hard-fails. |
|
|
| ## API |
|
|
| ### `POST /forecast` |
|
|
| ```jsonc |
| // request |
| { |
| "prices": [0.48, 0.50, 0.51, 0.52, ...], // historical yes_prices |
| "covariates": [[...], ...], // optional related series |
| "question_type": "event", // "event" | "numeric" |
| "horizon": 5, |
| "question": "Will X happen by 2027?" // optional, stored for calibration |
| } |
| ``` |
|
|
| ```jsonc |
| // response |
| { |
| "timesfm": { "forecast": [...], "trend": "up" }, |
| "chronos2": { "forecast": [...], "trend": "up" }, |
| "ensemble": { "forecast": [...], "trend": "up", "brier_estimate": 0.21 }, |
| "use_forecast": true, // false if event w/ no price history, or series < 10 pts |
| "warning": null, // e.g. "Series too short for reliable forecast" |
| "id": "…" // calibration row id |
| } |
| ``` |
|
|
| **Ensemble logic** (`ensemble.py`) |
|
|
| | Step | Rule | |
| |------|------| |
| | Length guard | `< 10` points → `use_forecast=false` + warning | |
| | Event blend | `0.5·TimesFM + 0.5·Chronos-2` (equal weight — both models show negative R² on binary event markets, so neither dominates) | |
| | Numeric blend | `0.4·TimesFM + 0.6·Chronos-2` (Chronos-2 wins on fev-bench) | |
| | Base-rate prior | if `|last − 0.5| ≤ 0.1`, pull ensemble **15% toward 0.5** | |
| | Extremise | only if confidence `> 0.70`: `0.5 + 1.3·(p − 0.5)` | |
|
|
| ### `GET /calibration` |
|
|
| Returns the realised **Brier score**, a **reliability curve** (decile bins of |
| predicted vs. observed frequency), and **directional accuracy** over all |
| resolved forecasts — so you can benchmark against superforecasters (~0.15 |
| Brier) over time. Data is stored in `db.sqlite`. |
|
|
| ### `POST /resolve` |
|
|
| ```jsonc |
| { "id": "<forecast id>", "outcome": 1.0 } // 1.0 = YES happened, 0.0 = NO |
| ``` |
|
|
| Records a market's actual outcome so it feeds the calibration scoring. |
|
|
| ### `GET /health` |
|
|
| Liveness + which models actually loaded. |
|
|
| ## LangGraph pipeline (Phase 3) |
|
|
| `pipeline.py` can run the full **SCAN → TRIAGE → SYNTHESISE** loop across |
| Polymarket, Kalshi, Metaculus, and Manifold for local CLI experiments. The |
| browser Pipeline does not trust that raw scan as its market source: the Next.js |
| proxy sends a classified `market_context` built by the Predict Trust Layer, and |
| the Python graph uses only exact/likely classified markets for consensus. |
|
|
| ```bash |
| python pipeline.py "Will UK interest rates fall below 4% by Jan 2027?" |
| ``` |
|
|
| At the checkpoint, press **Enter** to accept or type **`override 0.42`** to set |
| your own probability. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |------|---------| |
| | `main.py` | FastAPI app, startup model loading, endpoints | |
| | `models.py` | TimesFM / Chronos-2 wrappers with graceful fallback | |
| | `ensemble.py` | Blending, base-rate prior, extremisation (numpy-only, testable) | |
| | `calibration.py` | SQLite store + Brier / reliability / directional scoring | |
| | `pipeline.py` | LangGraph SCAN→TRIAGE→SYNTHESISE workflow | |
| | `requirements.txt` | Python dependencies | |
|
|
| ## Notes |
|
|
| * `FORECAST_PORT` overrides the default port (`8008`). |
| * The Next.js app calls `/forecast` only when `FORECAST_URL` is set and an exact/likely |
| Polymarket market with a CLOB token was selected. |
| * The Next.js caller defaults to `FORECAST_TIMEOUT_MS=90000` and |
| `CALIBRATION_TIMEOUT_MS=10000` (server-only env overrides, capped at 240 seconds) so model |
| quality and cold starts are not cut short by the old short timeout. |
| * Returned ensembles are supporting Momentum signals only; they do not blend into the headline |
| probability, and the app degrades gracefully when the service is offline. |
|
|
| ## Browser pipeline (no terminal) |
|
|
| The same SCAN → TRIAGE → ENSEMBLE → SYNTHESISE → CHECKPOINT loop is exposed over |
| HTTP so it can be driven entirely from the web app: |
|
|
| - `POST /pipeline/start` — `{ "question": "...", "question_type": "event", |
| "market_context": { ... } }` runs to the human checkpoint and returns the full |
| state plus a `thread_id`. `market_context` is supplied by the Next.js proxy; |
| with it present, related/rejected markets are provenance only and cannot drive |
| the Pipeline probability or price-history selection. |
| - `POST /pipeline/resume` — `{ "thread_id": "...", "resume": "" }` to accept, or |
| `{ "resume": "override 0.42" }` to override. |
|
|
| In the Next.js app open **`/pipeline`** (e.g. http://localhost:3000/pipeline): |
| type a question, watch the scan / triage / arbitrage / ensemble panels populate, |
| then **Approve** or **Override** the synthesised probability. Requires |
| `FORECAST_URL` to be set so the app can reach this service. |
|
|
