Real HRRR + true per-zone ISO-NE + 7-day rolling backtest from data repo

README.md

@@ -14,19 +14,21 @@ short_description: Real-time day-ahead demand forecasting for ISO New England
 
 # ⚡ Multi-Modal Deep Learning for Energy Demand Forecasting
 
-Live demo of two models from our CS-137 final project (Tufts, Spring 2026):
+Live demo of the trained CNN-Transformer baseline (1.75 M params) from our CS-137 final project (Tufts, Spring 2026), blended in a per-zone weighted ensemble with **Chronos-Bolt-mini** (Amazon, 21 M params, zero-shot on demand history).
 
-1. **Part 1 baseline** — CNN-Transformer (1.75 M params), reaches **5.24 % MAPE** with real HRRR weather on the 2022 self-eval slice.
-2. **Ensemble (Baseline ⊕ Chronos-Bolt-mini, zero-shot, per-zone α)** — adds the 21 M-param Amazon foundation model on demand history alone (no weather, no fine-tuning) and reaches **4.21 % MAPE** in offline evaluation.
+**All inputs are now fully real** — no synthetic weather, no proportionally-split system demand:
 
-## What it does
+- HRRR f00 weather analyses for the past 24 h (NOAA AWS S3, public)
+- HRRR f01..f24 forecast for the future 24 h (most recent long cycle ≤ T-2h)
+- True per-zone load from ISO-NE's public 5-minute zonal estimated load feed
+- Calendar features (deterministic from timestamps)
 
-1. **Real-time tab**: every click pulls the most recent 24 h of ISO New England system demand from the [EIA Open Data API](https://www.eia.gov/opendata/) (`respondent=ISNE`, `type=D`), splits it into the 8 ISO-NE zones via fixed proportions, and runs the chosen model on it. The Space holds a personal `EIA_API_KEY` as a Secret; if EIA is unreachable we fall through to an ISO-NE legacy endpoint and finally to a bundled 2022 sample.
-2. **Backtest tab**: 7 pre-computed daily forecasts (Dec 25–31, 2022 at 00:00 UTC) with all three models side-by-side and a per-zone MAPE table. The baseline curves there were computed on the Tufts HPC cluster with **real HRRR weather**, so this tab reaches the headline accuracy that the live tab can't get without weather inputs.
+Headline offline number: **5.24 % MAPE** baseline / **4.21 % MAPE** ensemble (with future analyses at training time — see disclosure in `about.md`). Live MAPE will be modestly worse because deployment substitutes HRRR forecasts for the future window.
 
-## ⚠ Demo limitation — synthetic weather inputs (live tab)
+## What it does
 
-The live tab substitutes **zeros** (training-mean weather in z-score space) for the baseline's weather raster channels because real-time HRRR isn't accessible from the Space. Calendar features (hour-of-day, day-of-week, month, holiday flag) and the recent demand pattern still drive the output, so the forecast shape is preserved, but absolute accuracy is lower than the cluster's 5.24 %. **Ensemble** mode largely closes the gap because Chronos-Bolt-mini doesn't need weather at all.
+1. **Real-time tab**: every click pulls real ISO-NE per-zone demand + real HRRR weather and runs the ensemble. Expect ~3-5 min on the very first click of a fresh Space (cold HRRR cache + Chronos load), then ~10-30 s on subsequent clicks within the same uptime session.
+2. **Backtest tab**: 7 daily forecasts on the most recent fully-published days, with full predict-vs-truth comparisons + per-zone MAPE table. Refreshed daily by a GitHub Actions cron in the [auxiliary data repo](https://github.com/jeffliulab/new-england-real-time-power-predict-data).
 
 ## Links
 
@@ -47,18 +49,25 @@ python app.py     # http://localhost:7860
 | File | Purpose |
 |---|---|
 | `app.py` | Gradio Blocks UI + Real-time / Backtest / About tabs |
-| `iso_ne_fetch.py` | Live demand fetch: EIA API → ISO-NE legacy → bundled CSV |
+| `iso_ne_fetch.py` | High-level demand fetcher: live ISO-NE 5-min → hourly + bundled CSV fallback + 30-day data-repo cache |
+| `iso_ne_zonal.py` | Low-level ISO-NE 5-minute zonal CSV fetcher (cookie-prime) |
+| `hrrr_fetch.py` | Real-time HRRR weather fetcher (Herbie + AWS S3 + KDTree-based regrid + `/tmp` cache) |
 | `calendar_features.py` | 44-d calendar one-hot encoder |
-| `model_utils.py` | Baseline + Chronos-Bolt-mini loading, inference, per-zone ensemble |
+| `model_utils.py` | Model loading + inference + Chronos ensemble |
 | `models/cnn_transformer_baseline.py` | Baseline architecture (1.75 M params) |
 | `checkpoints/best.pt` | Trained baseline weights (~20 MB) |
-| `checkpoints/norm_stats.pt` | z-score statistics for de-/normalization |
-| `assets/backtest_2022_last7d.json` | 7-day cached forecasts shown in the Backtest tab |
-| `assets/` | Figures shown in the *About* tab |
+| `checkpoints/norm_stats.pt` | z-score statistics (weather + energy) |
+| `assets/` | Figures shown in the *About* tab + bundled fallback samples |
+| `assets/backtest_fallback.json` | Last-known-good backtest snapshot (used if data repo unreachable) |
 | `about.md` | Demo explanation rendered in the UI |
+| `packages.txt` | apt-style packages: `libeccodes-dev`, `libeccodes-tools` (for cfgrib) |
 
-## Secrets
+## No secrets required
 
-| Name | Purpose |
-|---|---|
-| `EIA_API_KEY` | Personal EIA Open Data key for live ISO-NE demand. Free; register at https://www.eia.gov/opendata/register.php. Without this secret the Space still works — it just falls through to the ISO-NE legacy endpoint and (if that also fails) a bundled 2022 sample. |
+The Space pulls real data from public, no-auth endpoints:
+- ISO-NE: `https://www.iso-ne.com/transform/csv/fiveminuteestimatedzonalload?start=...&end=...` (with browser-cookie prime; see `iso_ne_zonal.py`)
+- HRRR: `s3://noaa-hrrr-bdp-pds/hrrr.{date}/conus/...` via the Herbie library
+
+The Backtest tab loads pre-built JSON from the auxiliary data repo
+[`new-england-real-time-power-predict-data`](https://github.com/jeffliulab/new-england-real-time-power-predict-data),
+also public; no auth needed.

about.md

@@ -1,48 +1,68 @@
 ## About this demo
 
-This Space runs two models from our CS-137 final project on **live ISO New England demand history**:
+This Space runs the trained CNN-Transformer baseline from our CS-137 final project on **fully real, live ISO New England inputs**, blended with **Chronos-Bolt-mini** (Amazon, 21 M params, Apache-2.0, zero-shot on demand history alone) in a per-zone weighted ensemble.
 
-1. **Baseline only** — the Part 1 CNN-Transformer (1.75 M params). Reaches **5.24 % MAPE** with real HRRR weather on the 2022 self-evaluation slice; in this Space the weather inputs are synthetic so accuracy is degraded.
-2. **Ensemble (Baseline + Chronos-Bolt-mini)** — late-fusion of the baseline with [Chronos-Bolt-mini](https://huggingface.co/amazon/chronos-bolt-mini) (Amazon, 21 M params, Apache-2.0), used **zero-shot on demand history only** — no weather, no fine-tuning. Reaches **4.21 % MAPE** on the same offline slice and is the recommended path for this demo.
+There are two tabs:
 
-The Model selector at the top of the page switches between them. The Real-time tab always issues a forecast for *now*; the Backtest tab shows 7 pre-computed forecasts over the last week of 2022 with cluster-quality (real-HRRR) baselines so you can see the headline accuracy.
+1. **Real-time forecast** — every click pulls the latest 24 h of demand and HRRR weather, plus a 24 h HRRR forecast cycle, and produces a 24-hour 8-zone prediction.
+2. **Backtest (last 7 days)** — 7 daily forecasts on the most recent 7 fully-published days, refreshed every day at 04:00 UTC by a GitHub Actions cron in [`new-england-real-time-power-predict-data`](https://github.com/jeffliulab/new-england-real-time-power-predict-data).
 
-### Real-time data path
+### What's real (everything)
 
-Each click on **Forecast next 24 h** pulls the most recent 24 hours of ISO-NE system demand from the [EIA Open Data API](https://www.eia.gov/opendata/) (`respondent=ISNE`, `type=D`). The Space holds my personal API key as a Secret named `EIA_API_KEY`; if EIA is unreachable we fall back to an ISO-NE legacy endpoint and finally to the bundled 2022 sample. The status line above the plots tells you which source served the request (`live (EIA)`, `live (ISO-NE)`, `cached`, or `sample-2022`).
+| Component | Source | Real or synthetic? |
+|---|---|---|
+| Per-zone demand history (24 h) | ISO-NE public 5-min `fiveminuteestimatedzonalload` feed → hourly mean | ✅ live (~1-2 h publication lag) |
+| Chronos context (720 h history) | Same ISO-NE feed (data repo cache + live splice) | ✅ live |
+| Weather history (24 h, 7 channels) | NOAA HRRR f00 analyses on AWS S3 (`noaa-hrrr-bdp-pds`) via Herbie | ✅ live |
+| Weather forecast (24 h, 7 channels) | NOAA HRRR cycle T-1's f01..f24 forecasts | ✅ live |
+| Calendar features | Computed deterministically from timestamps | ✅ |
+| Baseline weights | Trained on 2019–2022 data | ✅ |
+| Chronos-Bolt-mini | Amazon, zero-shot, no fine-tuning | ✅ |
 
-### What's real vs. synthetic
+The bundled 2022 sample CSVs are kept ONLY as a final fallback for when the live ISO-NE / HRRR endpoints are unreachable.
 
-| Component | Baseline only | Ensemble | Cluster runs |
-|---|---|---|---|
-| Baseline weights | ✅ | ✅ | ✅ |
-| Calendar features | ✅ | ✅ | ✅ |
-| Demand history | ✅ live ISO-NE (or 2022 fallback) | ✅ | ✅ |
-| **Weather inputs to baseline** | ❌ zeros (training-mean) | ❌ zeros (training-mean) | ✅ real HRRR rasters |
-| Chronos-Bolt-mini (zero-shot, demand only) | — | ✅ | — |
+### Strict-discipline backtest
 
-In Baseline-only mode, the forecast is degraded vs. the cluster's **5.24 %** MAPE because real weather is replaced with z-score zeros. Calendar features (hour, day-of-week, month, holiday flag) and the recent demand pattern still drive the output, so the shape of the forecast (daily double-peak, weekend/weekday differences) is preserved.
+For each daily forecast at time **T** (the last 7 days at 00:00 UTC each):
 
-In Ensemble mode, Chronos-Bolt-mini receives 720 hours (4 weeks) of recent per-zone demand and outputs a zero-shot 24-hour forecast for each zone. Per-zone weights $\alpha_z$ (shown beneath the chart) control the blend: $\alpha_z = 1$ keeps only the baseline; $\alpha_z = 0$ keeps only Chronos. The values come from a grid search on a 14-day validation window (2022-12-16 → 12-29) and are hard-coded in this Space — see Table 10 of the report for the underlying ablation.
+- **Demand history** for hours [T-24, T-1] comes from the public 5-min zonal feed
+- **Weather history** is 24 HRRR f00 analyses, one per valid hour [T-24, T-1]
+- **Weather forecast** is HRRR cycle (T-1)'s f01..f24 — i.e. the most recent forecast that was issued *before* T, with valid hours [T, T+23]
+- **Truth** for MAPE is the ISO-NE per-zone load for [T, T+23]
 
-### Backtest tab
+In particular **no future analyses are used** — every forecast at T sees only data that would have been available at time T, matching what a real deployment would do.
 
-The **Backtest** tab plays back 7 daily forecasts (Dec 25��31, 2022 at 00:00 UTC) from the `space/assets/backtest_2022_last7d.json` cache. The baseline curves there were computed on the Tufts HPC cluster with real HRRR weather inputs, so this tab demonstrates the headline accuracy that the live tab can't reach without weather. The Chronos and Ensemble curves are computed locally with the same code paths the live tab uses.
+### Disclosure: training-time `future_weather` mismatch
 
-### Per-zone allocation
+The trained baseline saw **HRRR f00 analyses for both history AND future windows during training** (i.e. the model was given the actual weather that occurred during the prediction window as an *input* feature). This is a form of supervised-learning-with-privileged-information that the report acknowledges in §4.1.5 / §5.
 
-ISO-NE's public data feed publishes *system-level* demand at hourly granularity. We split that total into 8 zones using fixed proportions estimated from 2022 historical zonal load reports. Per-zone real-time data requires an authenticated ISO Express account.
+At deployment we cannot use future analyses (they don't exist for the future yet), so we substitute HRRR forecasts (`f01..f24`) issued at the cycle just before the forecast time. The model therefore sees a slightly out-of-distribution input for the future window. **MAPE on this real-time / strict-backtest setup will be modestly worse than the offline 5.24 % headline** which used analyses for both windows.
 
-### What this is for
+This Space measures the deployable accuracy honestly. The Chronos-Bolt-mini ensemble path partially compensates because Chronos doesn't use weather at all.
 
-This is a **technical demonstration** of the trained models' input/output pipelines, not a production forecasting service. The full pipeline (live HRRR weather + authenticated per-zone real-time demand + the Chronos-Bolt-mini foundation-model ensemble) is documented in the report and tracked as future work in the GitHub repo.
+### Per-zone allocation — actually per-zone now
+
+Earlier prototypes of this demo used a fixed proportion vector to split the system total (from the EIA Open Data API) into 8 zones, which made the per-zone view cosmetic. The current Space pulls **true per-zone load** from ISO-NE's 5-minute estimated zonal feed, so per-zone forecasts are real.
 
 ### First-call latency
 
-If the **Ensemble** mode hasn't been used yet on this Space instance, the first request will trigger a one-time download of Chronos-Bolt-mini weights ($\sim$80 MB from HuggingFace Hub). Expect $\sim$30 s the first time and $\sim$5 s on subsequent requests. Baseline-only mode is always $\sim$2 s.
+The first Live tab click triggers:
+1. ~24 HRRR analysis cycles + 24 HRRR forecast hours from AWS S3 (parallel-fetched, cached at `/tmp/hrrr_cache/`)
+2. One Chronos-Bolt-mini load (~80 MB from HuggingFace Hub)
+
+Expect **~3-5 minutes on the very first click** of a fresh Space instance, and ~10-30 s on subsequent clicks within the same uptime session. The Backtest tab is instant — its data ships pre-computed from the data repo.
+
+### Per-zone ensemble weights
+
+Per-zone $\alpha_z$ (shown beneath the chart) blends baseline and Chronos:
+
+$$\hat y_z = \alpha_z \cdot \hat y_z^{\text{baseline}} + (1 - \alpha_z) \cdot \hat y_z^{\text{Chronos}}$$
+
+$\alpha_z$ values come from a grid search on a 14-day validation window in 2022. See Table 10 of the report for the underlying ablation.
 
 ### Links
 
 - 📄 [Final report (PDF)](https://github.com/jeffliulab/real-time-power-predict/blob/main/report/final_report.pdf)
-- 💻 [GitHub repository](https://github.com/jeffliulab/real-time-power-predict)
+- 💻 [Main code repo](https://github.com/jeffliulab/real-time-power-predict)
+- 🤖 [Auxiliary data repo (cron-refreshed backtest data)](https://github.com/jeffliulab/new-england-real-time-power-predict-data)
 - 👤 Author: **Pang Liu** · `pliu07` · Tufts CS-137

app.py

@@ -1,40 +1,41 @@
-"""Gradio Space: Multi-Modal Deep Learning for Energy Demand Forecasting.
-
-Real-time mode (always-now, no user-supplied datetime):
-  - Pulls the most recent 24 h of ISO-NE system demand from the EIA Open
-    Data API (free key, exposed to this Space as the `EIA_API_KEY`
-    secret), splits it into the 8 ISO-NE zones via fixed proportions,
-    and runs the chosen model on it.
-  - Falls back to the bundled 2022 sample window when the live API is
-    unreachable.
-
-Two model modes:
-  - Baseline only:  Part 1 CNN-Transformer (1.75 M params) on synthetic
-                    weather + real demand history.
-  - Ensemble (Baseline + Chronos-Bolt-mini): weather-aware baseline
-                    blended per-zone with the 21 M-param foundation
-                    model used zero-shot on demand history. Per Table 10
-                    of the report, mini gives the best per-zone ensemble
-                    (4.21 % test MAPE) and is small enough to run on the
-                    HF Spaces free CPU tier.
-
-Backtest tab:
-  - Pre-computed 7-day backtest (Dec 25-31, 2022) showing all three
-    models' forecasts vs. ground truth, with per-zone and overall MAPE.
-  - The baseline forecasts in this cache use REAL HRRR weather (computed
-    on the cluster), so this tab demonstrates the headline accuracy
-    that the live tab can't reach without weather inputs.
+"""Gradio Space: ISO-NE day-ahead demand forecasting (real-time + backtest).
+
+Always-now real-time forecast on truly real inputs:
+  - HRRR f00 weather analyses for the past 24 h (NOAA AWS S3, public)
+  - HRRR forecast cycle T-1's f01..f24 for the future 24 h (no future
+    analyses are used — strict deployable forecast)
+  - Per-zone ISO-NE 5-minute estimated zonal load, rolled up to hourly
+  - Calendar features (deterministic from timestamps)
+  - Chronos-Bolt-mini zero-shot foundation-model ensemble
+
+Backtest tab loads a 7-day rolling cache from the auxiliary data repo
+(``new-england-real-time-power-predict-data``), refreshed daily by a
+GitHub Actions cron. Cache is fetched once at Space startup; falls back
+to a bundled snapshot if the data repo is unreachable.
+
+Disclosure (also in about.md): the trained baseline saw f00 ANALYSES
+for both history AND future windows during training (a form of data
+leakage). At deployment we substitute HRRR f01..f24 forecasts for the
+future window — there is no future-data leak, but the model sees a
+slightly out-of-distribution input. Live MAPE will therefore be a bit
+worse than the offline 5.24 % headline.
 """
 
 from __future__ import annotations
 
 import json
+import os
+import shutil
+import time
 from datetime import datetime, timedelta, timezone
 from pathlib import Path
+from typing import Optional
 
 import gradio as gr
 import numpy as np
+import pandas as pd
 import plotly.graph_objects as go
+import requests
 from plotly.subplots import make_subplots
 
 from calendar_features import encode_range
@@ -46,12 +47,23 @@ from model_utils import (
     run_forecast,
     per_zone_ensemble,
     ALPHA_PER_ZONE_MINI,
+    HISTORY_LEN,
+    FUTURE_LEN,
+)
+from hrrr_fetch import (
+    fetch_history as hrrr_fetch_history,
+    fetch_forecast_for_window,
 )
 
 ROOT = Path(__file__).parent
 ASSETS = ROOT / "assets"
 ABOUT = (ROOT / "about.md").read_text()
-BACKTEST_JSON = ASSETS / "backtest_2022_last7d.json"
+
+DATA_REPO_BASE = "https://raw.githubusercontent.com/jeffliulab/new-england-real-time-power-predict-data/main"
+BACKTEST_URL = f"{DATA_REPO_BASE}/data/backtest_rolling_7d.json"
+THIRTY_DAY_CSV_URL = f"{DATA_REPO_BASE}/data/iso_ne_30d.csv"
+LAST_BUILT_URL = f"{DATA_REPO_BASE}/data/last_built.json"
+THIRTY_DAY_CACHE_PATH = Path("/tmp/iso_ne_30d.csv")
 
 NAVY = "#1A3A5C"
 ACCENT = "#2E86DE"
@@ -63,7 +75,7 @@ print("Loading baseline checkpoint...")
 MODEL, NORM_STATS = load_baseline(ROOT / "checkpoints" / "best.pt", device="cpu")
 print(f"Loaded baseline ({sum(p.numel() for p in MODEL.parameters()):,} params)")
 
-# Lazy-loaded Chronos pipeline (only when the user picks the ensemble model).
+# Lazy-loaded Chronos pipeline (loaded on first Live forecast click)
 _CHRONOS = {"pipeline": None}
 
 
@@ -75,67 +87,163 @@ def _get_chronos():
     return _CHRONOS["pipeline"]
 
 
+def _bootstrap_data_repo():
+    """At startup, fetch the latest backtest JSON + 30-day CSV from the
+    auxiliary data repo. Saves the CSV to /tmp so iso_ne_fetch can find it.
+    Returns (backtest_dict, last_built_dict) or (None, None) if data repo
+    unreachable (Space falls back to bundled snapshot)."""
+    backtest = None
+    last_built = None
+    try:
+        r = requests.get(BACKTEST_URL, timeout=15)
+        r.raise_for_status()
+        backtest = r.json()
+        print(f"Loaded backtest JSON from data repo: "
+              f"{backtest.get('n_forecasts')} forecasts, "
+              f"built_at={backtest.get('built_at')}")
+    except Exception as e:  # noqa: BLE001
+        print(f"WARN: failed to fetch backtest JSON ({e}); will use bundled fallback")
+
+    try:
+        r = requests.get(LAST_BUILT_URL, timeout=10)
+        r.raise_for_status()
+        last_built = r.json()
+    except Exception as e:  # noqa: BLE001
+        print(f"WARN: failed to fetch last_built metadata ({e})")
+
+    try:
+        r = requests.get(THIRTY_DAY_CSV_URL, timeout=20)
+        r.raise_for_status()
+        THIRTY_DAY_CACHE_PATH.write_bytes(r.content)
+        print(f"Cached 30d CSV at {THIRTY_DAY_CACHE_PATH} "
+              f"({len(r.content) / 1024:.1f} KB)")
+    except Exception as e:  # noqa: BLE001
+        print(f"WARN: failed to fetch 30d CSV ({e}); Chronos context will use bundled sample")
+
+    return backtest, last_built
+
+
+BACKTEST, LAST_BUILT = _bootstrap_data_repo()
+if BACKTEST is None:
+    # Fallback to bundled snapshot if it exists (shipped with the Space)
+    fallback = ASSETS / "backtest_fallback.json"
+    if fallback.exists():
+        try:
+            BACKTEST = json.loads(fallback.read_text())
+            print("Using bundled backtest_fallback.json")
+        except Exception as e:  # noqa: BLE001
+            print(f"WARN: bundled fallback also failed: {e}")
+
+
 def _now_utc_hour() -> datetime:
-    return datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
+    return datetime.now(timezone.utc).replace(
+        minute=0, second=0, microsecond=0, tzinfo=None)
 
 
 # =====================================================================
-#  Real-time forecast
+#  Live forecast (real-time)
 # =====================================================================
 
-def forecast(model_choice: str):
-    """Always-now real-time forecast: pulls live demand, runs the chosen model."""
+def live_forecast(progress: Optional[gr.Progress] = None):
+    """Pull real HRRR + real ISO-NE per-zone, run baseline + Chronos
+    ensemble, and return plots + summary markdown.
+
+    Uses Gradio's Progress widget for the slow HRRR fetch step.
+    """
+    progress = progress or gr.Progress()
+
     target = _now_utc_hour()
-    hist_start = target - timedelta(hours=24)
-
-    hist_demand, source = fetch_recent_demand_mwh(target)
-    hist_cal = encode_range(hist_start, 24)
-    fut_cal = encode_range(target, 24)
-
-    baseline_pred = run_forecast(MODEL, hist_demand, hist_cal, fut_cal,
-                                  NORM_STATS, device="cpu")
-
-    if model_choice == "Ensemble (Baseline + Chronos-Bolt-mini)":
-        long_history, long_source = fetch_long_history_mwh(target, hours=720)
-        pipeline = _get_chronos()
-        chronos_pred = run_chronos_zeroshot(pipeline, long_history)
-        pred_mwh = per_zone_ensemble(baseline_pred, chronos_pred,
-                                      ALPHA_PER_ZONE_MINI)
-        line = _line_plot(target, hist_demand, pred_mwh,
-                           overlay={"Baseline only": baseline_pred,
-                                    "Chronos-Bolt-mini only": chronos_pred})
-        active_label = (
-            f"**Ensemble** (Baseline ⊕ Chronos-Bolt-mini, per-zone α). "
-            f"Demand source for Chronos: `{long_source}`."
-        )
-    else:
-        pred_mwh = baseline_pred
-        line = _line_plot(target, hist_demand, pred_mwh)
-        active_label = "**Baseline only** (Part 1 CNN-Transformer, synthetic weather)."
-
-    bar = _bar_plot(target, pred_mwh[0])
-    sys_total = pred_mwh.sum(axis=1)
+
+    progress(0.05, desc="Fetching ISO-NE per-zone demand...")
+    try:
+        hist_demand, demand_src = fetch_recent_demand_mwh(target)
+    except Exception as e:  # noqa: BLE001
+        return _error_panel(f"ISO-NE demand fetch failed: {e}")
+
+    progress(0.10, desc="Fetching HRRR weather history (24 cycles)...")
+    fetched = {"count": 0}
+    def _hist_progress(done, total, label):
+        fetched["count"] = done
+        progress(0.10 + 0.40 * done / total,
+                 desc=f"HRRR history {done}/{total} — {label}")
+    try:
+        hist_w_raw = hrrr_fetch_history(target, hours=HISTORY_LEN,
+                                          parallel=4,
+                                          progress=_hist_progress)
+    except Exception as e:  # noqa: BLE001
+        return _error_panel(f"HRRR history fetch failed: {e}")
+
+    progress(0.55, desc="Fetching HRRR weather forecast (latest long cycle)...")
+    def _fut_progress(done, total, label):
+        progress(0.55 + 0.20 * done / total,
+                 desc=f"HRRR forecast {done}/{total} — {label}")
+    try:
+        fut_w_raw, cycle_for_future, fxx_start = fetch_forecast_for_window(
+            target, hours=FUTURE_LEN, parallel=4,
+            progress=_fut_progress)
+    except Exception as e:  # noqa: BLE001
+        return _error_panel(f"HRRR forecast fetch failed: {e}")
+
+    progress(0.80, desc="Running baseline forward pass...")
+    hist_cal = encode_range(target - timedelta(hours=HISTORY_LEN), HISTORY_LEN)
+    fut_cal = encode_range(target, FUTURE_LEN)
+    try:
+        baseline_pred = run_forecast(
+            MODEL, hist_demand, hist_cal, fut_cal, NORM_STATS,
+            hist_weather_raw=hist_w_raw, future_weather_raw=fut_w_raw,
+            device="cpu")
+    except Exception as e:  # noqa: BLE001
+        return _error_panel(f"Baseline forward failed: {e}")
+
+    progress(0.88, desc="Running Chronos-Bolt-mini zero-shot...")
+    try:
+        long_history, long_src = fetch_long_history_mwh(target, hours=720)
+        chronos_pipeline = _get_chronos()
+        chronos_pred = run_chronos_zeroshot(chronos_pipeline, long_history)
+    except Exception as e:  # noqa: BLE001
+        return _error_panel(f"Chronos forecast failed: {e}")
+
+    progress(0.95, desc="Computing ensemble + plotting...")
+    ens_pred = per_zone_ensemble(baseline_pred, chronos_pred, ALPHA_PER_ZONE_MINI)
+
+    line = _live_line_plot(target, hist_demand, ens_pred,
+                            overlay={"Baseline (with HRRR)": baseline_pred,
+                                     "Chronos zero-shot": chronos_pred})
+    bar = _live_bar_plot(target, ens_pred[0])
+
+    sys_total = ens_pred.sum(axis=1)
     summary = (
-        f"{active_label}  \n"
-        f"Demand history source: `{source}`  ·  "
-        f"forecast issued at **{target.strftime('%Y-%m-%d %H:00')} UTC**  ·  "
-        f"covers next 24 h to **{(target + timedelta(hours=24)).strftime('%Y-%m-%d %H:00')} UTC**  ·  "
-        f"system-level peak: **{sys_total.max():,.0f} MW**."
+        f"### Forecast issued at **{target.strftime('%Y-%m-%d %H:00')} UTC**\n\n"
+        f"**Inputs**\n"
+        f"- Demand history: `{demand_src}`\n"
+        f"- Chronos context: `{long_src}`\n"
+        f"- Weather history: real HRRR f00 analyses, "
+        f"24 cycles {(target - timedelta(hours=24)).strftime('%Y-%m-%d %H:00')} → "
+        f"{(target - timedelta(hours=1)).strftime('%H:00')} UTC\n"
+        f"- Weather forecast: real HRRR cycle "
+        f"{cycle_for_future.strftime('%Y-%m-%d %H:00')} UTC, "
+        f"f{fxx_start:02d}..f{fxx_start + FUTURE_LEN - 1:02d}\n\n"
+        f"**Output**: 24-hour ensemble forecast covering "
+        f"**{target.strftime('%H:00')} → {(target + timedelta(hours=24)).strftime('%H:00')} UTC** · "
+        f"system-level peak: **{sys_total.max():,.0f} MW**"
     )
+    progress(1.0, desc="Done")
     return line, bar, summary
 
 
-def _line_plot(target: datetime, hist: np.ndarray, pred: np.ndarray,
-                overlay: dict[str, np.ndarray] | None = None):
-    """4 subplots * 2 zones each, each showing history + forecast (+ optional overlays)."""
+def _error_panel(msg: str):
+    return go.Figure(), go.Figure(), f"### ⚠ Forecast failed\n\n{msg}"
+
+
+def _live_line_plot(target: datetime, hist: np.ndarray, pred: np.ndarray,
+                    overlay: dict[str, np.ndarray]):
+    """8 panels (4×2). History solid + ensemble dashed; overlays as dotted."""
     fig = make_subplots(rows=4, cols=2, shared_xaxes=False,
-                        subplot_titles=ZONE_COLS,
-                        vertical_spacing=0.10, horizontal_spacing=0.07)
+                          subplot_titles=ZONE_COLS,
+                          vertical_spacing=0.10, horizontal_spacing=0.07)
     hist_t = [target - timedelta(hours=24 - i) for i in range(24)]
     fut_t = [target + timedelta(hours=i + 1) for i in range(24)]
-    overlay = overlay or {}
     overlay_palette = [GREY, TEAL, AMBER]
-
     for i, zone in enumerate(ZONE_COLS):
         r, c = i // 2 + 1, i % 2 + 1
         fig.add_trace(go.Scatter(
@@ -146,18 +254,17 @@ def _line_plot(target: datetime, hist: np.ndarray, pred: np.ndarray,
         fig.add_trace(go.Scatter(
             x=fut_t, y=pred[:, i], mode="lines",
             line=dict(color=ACCENT, width=2.5, dash="dash"),
-            name="forecast (active)", showlegend=(i == 0),
+            name="forecast (ensemble)", showlegend=(i == 0),
         ), row=r, col=c)
         for k, (label, arr) in enumerate(overlay.items()):
-            colour = overlay_palette[k % len(overlay_palette)]
             fig.add_trace(go.Scatter(
                 x=fut_t, y=arr[:, i], mode="lines",
-                line=dict(color=colour, width=1.2, dash="dot"),
+                line=dict(color=overlay_palette[k], width=1.2, dash="dot"),
                 name=label, showlegend=(i == 0),
                 opacity=0.85,
             ), row=r, col=c)
         fig.add_vline(x=target, line=dict(color="grey", width=1, dash="dot"),
-                      row=r, col=c)
+                       row=r, col=c)
     fig.update_layout(
         title="Per-zone demand: history (solid) and 24-h forecast (dashed)",
         height=820, plot_bgcolor="white",
@@ -169,8 +276,7 @@ def _line_plot(target: datetime, hist: np.ndarray, pred: np.ndarray,
     return fig
 
 
-def _bar_plot(target: datetime, next_hour_pred: np.ndarray):
-    """Horizontal bar: predicted demand at target+1h, sorted."""
+def _live_bar_plot(target: datetime, next_hour_pred: np.ndarray):
     order = np.argsort(next_hour_pred)
     fig = go.Figure(go.Bar(
         x=next_hour_pred[order], y=[ZONE_COLS[i] for i in order],
@@ -179,7 +285,8 @@ def _bar_plot(target: datetime, next_hour_pred: np.ndarray):
         textposition="outside",
     ))
     fig.update_layout(
-        title=f"Predicted demand at t+1h ({(target + timedelta(hours=1)).strftime('%Y-%m-%d %H:00')} UTC)",
+        title=f"Predicted demand at t+1h "
+              f"({(target + timedelta(hours=1)).strftime('%Y-%m-%d %H:00')} UTC)",
         xaxis_title="MW", height=350, plot_bgcolor="white",
         margin=dict(l=80, r=40, t=60, b=40),
     )
@@ -192,28 +299,20 @@ def _alpha_table_md() -> str:
 
 
 # =====================================================================
-#  Backtest tab (cached: 7 forecasts, Dec 25-31, 2022, real HRRR weather)
+#  Backtest tab (rolling 7-day, loaded at startup from data repo)
 # =====================================================================
 
-if BACKTEST_JSON.exists():
-    BACKTEST = json.loads(BACKTEST_JSON.read_text())
-else:
-    BACKTEST = None
-    print(f"WARNING: backtest cache not found at {BACKTEST_JSON}")
-
-
 def _backtest_overview_plot():
-    """One row per zone, showing 7-day truth vs. each model's forecast."""
     if BACKTEST is None:
         return go.Figure()
     forecasts = BACKTEST["forecasts"]
     fig = make_subplots(rows=4, cols=2, shared_xaxes=False,
-                        subplot_titles=ZONE_COLS,
-                        vertical_spacing=0.10, horizontal_spacing=0.07)
+                          subplot_titles=ZONE_COLS,
+                          vertical_spacing=0.10, horizontal_spacing=0.07)
     for i, zone in enumerate(ZONE_COLS):
         r, c = i // 2 + 1, i % 2 + 1
         for f in forecasts:
-            start = datetime.fromisoformat(f["start"]).replace(tzinfo=timezone.utc)
+            start = datetime.fromisoformat(f["start"])
             t = [start + timedelta(hours=h) for h in range(24)]
             truth = np.asarray(f["truth_24h"])[:, i]
             base = np.asarray(f["baseline"])[:, i]
@@ -242,12 +341,16 @@ def _backtest_overview_plot():
                 line=dict(color=ACCENT, width=2, dash="dash"),
                 name="ensemble", showlegend=show,
             ), row=r, col=c)
+    period = BACKTEST.get("data_period", {})
     fig.update_layout(
-        title="7-day backtest, Dec 25-31 2022 — actual demand vs. 3 model variants",
+        title=(f"7-day rolling backtest "
+               f"({period.get('first_forecast_start', '?')[:10]} → "
+               f"{period.get('last_forecast_start', '?')[:10]}) "
+               f"— actual vs 3 model variants"),
         height=900, plot_bgcolor="white",
         margin=dict(l=40, r=20, t=80, b=40),
         legend=dict(orientation="h", yanchor="bottom", y=1.02,
-                    xanchor="right", x=1),
+                     xanchor="right", x=1),
     )
     fig.update_yaxes(title_text="MW", title_standoff=4)
     return fig
@@ -255,36 +358,45 @@ def _backtest_overview_plot():
 
 def _backtest_summary_md() -> str:
     if BACKTEST is None:
-        return "_Backtest cache missing — re-run `scripts/build_space_backtest.py`._"
+        return ("_Rolling backtest unavailable — auxiliary data repo unreachable_\n\n"
+                "The Backtest tab loads its data from "
+                "[`new-england-real-time-power-predict-data`]"
+                "(https://github.com/jeffliulab/new-england-real-time-power-predict-data) "
+                "which a GitHub Actions cron refreshes every day.")
     s = BACKTEST["summary"]
+    period = BACKTEST.get("data_period", {})
     rows = []
     rows.append("| Model | " + " | ".join(ZONE_COLS) + " | **Overall** |")
     rows.append("|---|" + "|".join(["---"] * (len(ZONE_COLS) + 1)) + "|")
     for key, label in (("baseline", "Baseline (real HRRR)"),
-                       ("chronos",  "Chronos-Bolt-mini (zero-shot)"),
-                       ("ensemble", "Ensemble (per-zone α)")):
+                        ("chronos",  "Chronos-Bolt-mini (zero-shot)"),
+                        ("ensemble", "Ensemble (per-zone α)")):
         per_zone = " | ".join(f"{s[key]['per_zone'][z]:.2f}" for z in ZONE_COLS)
         rows.append(f"| {label} | {per_zone} | **{s[key]['overall']:.2f}** |")
     table = "\n".join(rows)
+    built_at = BACKTEST.get("built_at", "?")
     return (
-        f"### 7-day average MAPE (%) over {BACKTEST['n_forecasts']} forecasts (Dec 25–31, 2022)\n\n"
+        f"### Last 7 days of forecasts — per-zone & overall MAPE (%)\n\n"
+        f"_Window: {period.get('first_forecast_start', '?')[:16]} UTC → "
+        f"{period.get('last_forecast_start', '?')[:16]} UTC · "
+        f"refreshed {built_at[:16]} UTC_\n\n"
         f"{table}\n\n"
-        f"_Each forecast is a 24-hour prediction starting at 00:00 UTC. The "
-        f"baseline numbers in this table use **real HRRR weather** (computed "
-        f"on the cluster), so they reflect the headline 5.24 % test MAPE setup. "
-        f"The live tab above uses synthetic weather, so its accuracy is lower; "
-        f"the **Ensemble** path closes most of that gap because Chronos-Bolt-mini "
-        f"doesn't need weather at all._"
+        f"_Each forecast issues a 24-hour prediction at 00:00 UTC. The baseline uses "
+        f"real HRRR f00 analyses for the history window (24 cycles) and HRRR f01..f24 "
+        f"forecasts from the cycle issued at T-1 for the future window — strict deployable "
+        f"backtest with no future-data leak. See **About** for the disclosure on the "
+        f"training-time future_weather mismatch._"
     )
 
 
 def _backtest_bars():
-    """Bar chart: overall MAPE per model (averaged over 7 forecasts)."""
     if BACKTEST is None:
         return go.Figure()
     s = BACKTEST["summary"]
-    labels = ["Baseline\n(real HRRR)", "Chronos-Bolt-mini\n(zero-shot)", "Ensemble\n(per-zone α)"]
-    values = [s["baseline"]["overall"], s["chronos"]["overall"], s["ensemble"]["overall"]]
+    labels = ["Baseline\n(real HRRR)", "Chronos-Bolt-mini\n(zero-shot)",
+              "Ensemble\n(per-zone α)"]
+    values = [s["baseline"]["overall"], s["chronos"]["overall"],
+              s["ensemble"]["overall"]]
     fig = go.Figure(go.Bar(
         x=labels, y=values, marker_color=[GREY, TEAL, ACCENT],
         text=[f"{v:.2f}%" for v in values], textposition="outside",
@@ -298,54 +410,42 @@ def _backtest_bars():
 
 
 # =====================================================================
-#  Gradio layout
+#  Gradio UI
 # =====================================================================
 
 with gr.Blocks(title="ISO-NE Energy Demand Forecast",
-               theme=gr.themes.Default(primary_hue="blue")) as demo:
+                 theme=gr.themes.Default(primary_hue="blue")) as demo:
     gr.Markdown(
         "# ⚡ Multi-Modal Deep Learning for Energy Demand Forecasting\n"
         "**Author:** Pang Liu · Tufts CS-137 · "
         "[GitHub](https://github.com/jeffliulab/real-time-power-predict)\n\n"
-        "> 🔴 **Real-time mode**: every click pulls the most recent ISO-NE system "
-        "demand from the EIA Open Data API and forecasts the next 24 h.\n"
-        "> ⚠ **Demo limitation**: weather inputs are synthetic (training-mean "
-        "zeros) since real-time HRRR rasters aren't available in this Space. "
-        "The cluster runs reach **5.24 % MAPE** with real HRRR weather; the "
-        "**Ensemble** path adds Chronos-Bolt-mini (zero-shot on demand history "
-        "only — no weather) and reaches **4.21 % MAPE** in our offline "
-        "evaluation. See the **Backtest** tab for a 7-day side-by-side "
-        "comparison and the **About** tab for full details."
+        "Live tab pulls real ISO-NE per-zone demand + real HRRR weather "
+        "(history analyses + forecast-cycle predictions) and runs the trained "
+        "CNN-Transformer baseline blended with Chronos-Bolt-mini in a per-zone "
+        "weighted ensemble. The Backtest tab shows the same model on the most "
+        "recent 7 fully-published days, refreshed daily by GitHub Actions cron "
+        "in the auxiliary data repo."
     )
     with gr.Row():
-        model_choice = gr.Radio(
-            choices=["Baseline only",
-                     "Ensemble (Baseline + Chronos-Bolt-mini)"],
-            value="Ensemble (Baseline + Chronos-Bolt-mini)",
-            label="Model",
-            scale=2,
-        )
-        run_btn = gr.Button("Forecast next 24 h (now)",
-                             variant="primary", scale=1)
+        run_btn = gr.Button("🔮 Forecast next 24 h (now)",
+                              variant="primary", scale=1, size="lg")
     summary_md = gr.Markdown()
     with gr.Tabs():
         with gr.Tab("Real-time forecast"):
             line_plot = gr.Plot(label="Per-zone history + forecast")
             bar_plot = gr.Plot(label="Predicted next-hour demand")
             gr.Markdown(_alpha_table_md())
-        with gr.Tab("Backtest (last 7 days of 2022)"):
+        with gr.Tab("Backtest (last 7 days)"):
             gr.Markdown(
-                "These are 7 daily forecasts on the held-out 2022-12-25 → "
-                "12-31 window, each issued at 00:00 UTC for the next 24 h. "
-                "The **baseline** column uses real HRRR weather (computed "
-                "offline on the cluster); **Chronos-Bolt-mini** is zero-shot; "
-                "the **ensemble** is the per-zone weighted blend reported in "
-                "the paper."
+                "_Strict-discipline backtest_ — at each forecast time T the "
+                "model sees only data available before T. History weather: "
+                "24 HRRR f00 analyses; future weather: f01..f24 from cycle "
+                "T-1 (the most recent cycle issued before T)."
             )
             backtest_plot = gr.Plot(value=_backtest_overview_plot(),
-                                     label="7-day per-zone comparison")
+                                      label="7-day per-zone comparison")
             backtest_bars = gr.Plot(value=_backtest_bars(),
-                                     label="Overall MAPE")
+                                      label="Overall MAPE")
             gr.Markdown(_backtest_summary_md())
         with gr.Tab("About"):
             gr.Markdown(ABOUT)
@@ -360,10 +460,8 @@ with gr.Blocks(title="ISO-NE Energy Demand Forecast",
                      label="Baseline CNN-Transformer architecture",
                      show_label=True)
 
-    run_btn.click(forecast, inputs=[model_choice],
+    run_btn.click(live_forecast,
                   outputs=[line_plot, bar_plot, summary_md])
-    demo.load(forecast, inputs=[model_choice],
-              outputs=[line_plot, bar_plot, summary_md])
 
 
 if __name__ == "__main__":

hrrr_fetch.py

@@ -0,0 +1,363 @@
+"""
+Real-time HRRR weather fetcher for the predict-power Space.
+
+This is the runtime counterpart to ``scripts/data_preparation/fetch_hrrr_weather.py``
+(used to build the training set). It MUST produce arrays in the same
+shape, channel order, and grid as training, otherwise the model sees an
+out-of-distribution input. Specifically:
+
+  - 7 channels in fixed order:
+      [TMP_2m, RH_2m, UGRD_10m, VGRD_10m, GUST_surface, DSWRF_surface, APCP_1hr]
+  - NE bbox: lat 40.5-47.5 N, lon -74.0 to -66.0 (West)
+  - Regridded to 450 lat-rows x 449 lon-cols via xarray.interp(linear),
+    NOT direct slicing of the native Lambert-Conformal grid
+
+We fetch from the public ``noaa-hrrr-bdp-pds`` AWS S3 bucket via the
+Herbie library (proven path; same as training).
+
+Two top-level entry points:
+  - ``fetch_history(end_dt, hours=24)`` returns ``(hours, 450, 449, 7)``,
+    one f00 analysis per requested hour
+  - ``fetch_forecast(cycle_dt, hours=24)`` returns ``(hours, 450, 449, 7)``,
+    cycle_dt's f01..f{hours} forecast hours
+
+Both paths are cached at ``/tmp/hrrr_cache/{cycle_YYYYMMDDHH}_f{NN}.npz``.
+The cache survives within an HF Space uptime session and is wiped on sleep.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime, timedelta, timezone
+from pathlib import Path
+from typing import Callable, Iterable, Optional, Sequence
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+# === Match training pipeline EXACTLY ===
+_BBOX = {"lat_min": 40.5, "lat_max": 47.5,
+         "lon_min": -74.0, "lon_max": -66.0}
+GRID_H = 450     # lat rows
+GRID_W = 449     # lon cols
+N_CHANNELS = 7
+
+# Target lat/lon grid (geographic, not native HRRR Lambert-Conformal)
+_LAT = np.linspace(_BBOX["lat_min"], _BBOX["lat_max"], GRID_H)
+_LON = np.linspace(_BBOX["lon_min"], _BBOX["lon_max"], GRID_W)
+
+# Channel definitions: (name, herbie searchString)
+_CHANNELS: list[tuple[str, str]] = [
+    ("TMP",      ":TMP:2 m above ground"),
+    ("RH",       ":RH:2 m above ground"),
+    ("UGRD",     ":UGRD:10 m above ground"),
+    ("VGRD",     ":VGRD:10 m above ground"),
+    ("GUST",     ":GUST:surface"),
+    ("DSWRF",    ":DSWRF:surface"),
+    ("APCP_1hr", ":APCP:surface:0-1 hour acc"),
+]
+
+CACHE_DIR = Path(os.environ.get("HRRR_CACHE_DIR", "/tmp/hrrr_cache"))
+CACHE_DIR.mkdir(parents=True, exist_ok=True)
+
+
+def _cache_path(cycle_dt: datetime, fxx: int) -> Path:
+    return CACHE_DIR / f"{cycle_dt.strftime('%Y%m%d%H')}_f{fxx:02d}.npz"
+
+
+def _hour_floor_utc(dt: datetime) -> datetime:
+    if dt.tzinfo is None:
+        dt = dt.replace(tzinfo=timezone.utc)
+    dt = dt.astimezone(timezone.utc)
+    return dt.replace(minute=0, second=0, microsecond=0, tzinfo=None)
+
+
+# --- regridding weights (computed lazily, then cached for the process) ---
+# HRRR's native Lambert-Conformal grid is fixed across cycles, so we can
+# precompute (mask, kdtree, weights, idxs) once from any sample dataset.
+# Per-channel regrid is then a single matmul (~10 ms on cpu-basic).
+_REGRID_CACHE: dict = {}
+
+
+def _build_regrid_weights(lat2d: np.ndarray, lon2d_signed: np.ndarray):
+    """Build cropping mask + 4-NN inverse-distance weights for our target grid.
+
+    Returns dict with keys:
+      - ``mask``: bool array (1059, 1799) selecting cells inside an NE
+         bounding box that contains our target grid with ~1° margin
+      - ``idxs``: (450*449, 4) int32 — indices into the masked source array
+      - ``weights``: (450*449, 4) float32 — sums to 1 along axis=1
+    """
+    from scipy.spatial import cKDTree   # noqa: WPS433
+
+    # Crop with margin so target-grid corners always have neighbors in source
+    mask = ((lat2d >= _BBOX["lat_min"] - 1.5)
+            & (lat2d <= _BBOX["lat_max"] + 1.5)
+            & (lon2d_signed >= _BBOX["lon_min"] - 1.5)
+            & (lon2d_signed <= _BBOX["lon_max"] + 1.5))
+    if mask.sum() == 0:
+        raise RuntimeError("Bounding-box mask is empty; HRRR grid mismatch?")
+
+    src_pts = np.stack(
+        [lat2d[mask].astype(np.float64),
+         lon2d_signed[mask].astype(np.float64)],
+        axis=-1)
+    LL, LN = np.meshgrid(_LAT, _LON, indexing="ij")
+    tgt_pts = np.stack([LL.ravel(), LN.ravel()], axis=-1)
+
+    tree = cKDTree(src_pts)
+    dists, idxs = tree.query(tgt_pts, k=4)
+    # Inverse-distance weights, normalized
+    inv_d = 1.0 / np.maximum(dists, 1e-9)
+    w = (inv_d / inv_d.sum(axis=1, keepdims=True)).astype(np.float32)
+    return {"mask": mask, "idxs": idxs.astype(np.int32), "weights": w}
+
+
+def _regrid(field2d: np.ndarray, weights_pack: dict) -> np.ndarray:
+    """Apply precomputed mask + weights to a (1059, 1799) HRRR field, return
+    (450, 449) float32 on the regular lat/lon target grid."""
+    cropped = field2d[weights_pack["mask"]].astype(np.float32)
+    out = (cropped[weights_pack["idxs"]] * weights_pack["weights"]).sum(axis=1)
+    return out.reshape(GRID_H, GRID_W)
+
+
+def _fetch_one_via_herbie(cycle_dt: datetime, fxx: int) -> np.ndarray:
+    """Fetch one (cycle, forecast-hour) pair, return (450, 449, 7) float32.
+
+    Caller is responsible for caching; this function always hits the network.
+    Raises RuntimeError on any failure.
+    """
+    try:
+        from herbie import Herbie       # noqa: WPS433  (optional heavy dep)
+    except ImportError as e:
+        raise RuntimeError(
+            f"hrrr_fetch.py requires herbie-data: {e}") from e
+
+    H = Herbie(
+        cycle_dt.strftime("%Y-%m-%d %H:00"),
+        model="hrrr",
+        product="sfc",
+        fxx=fxx,
+        verbose=False,
+    )
+    channels: list[np.ndarray] = []
+    for ch_name, regex in _CHANNELS:
+        try:
+            # Newer Herbie (>=2024.x) renamed `searchString` to `search`
+            ds = H.xarray(search=regex, verbose=False)
+        except Exception as e:  # noqa: BLE001
+            # APCP accumulation window varies with forecast hour:
+            # f00 has no APCP, f01 has "0-1 hour acc" (matches our regex),
+            # f02 has "0-2 hour acc" or "1-2 hour acc", etc. We zero-fill
+            # any APCP fetch failure (the training mean is near zero in
+            # MM units anyway, so post-z-score the model sees ~0).
+            if ch_name == "APCP_1hr":
+                logger.info("APCP_1hr unavailable at %s f%02d (%s); using zero",
+                             cycle_dt, fxx,
+                             type(e).__name__ if not str(e) else str(e)[:80])
+                channels.append(np.zeros((GRID_H, GRID_W), dtype=np.float32))
+                continue
+            raise RuntimeError(
+                f"Herbie xarray() failed for {ch_name} at "
+                f"{cycle_dt.isoformat()} f{fxx:02d}: {e}") from e
+        var = list(ds.data_vars)[0]
+        arr = ds[var]
+        field2d = np.squeeze(arr.values)
+        if field2d.shape != (1059, 1799):
+            raise RuntimeError(
+                f"unexpected HRRR field shape {field2d.shape} for {ch_name}")
+
+        # Initialize regrid weights once per process from the first dataset
+        if "weights_pack" not in _REGRID_CACHE:
+            lat2d = arr.coords["latitude"].values
+            lon2d = arr.coords["longitude"].values
+            lon2d_signed = np.where(lon2d > 180, lon2d - 360, lon2d)
+            _REGRID_CACHE["weights_pack"] = _build_regrid_weights(
+                lat2d, lon2d_signed)
+            logger.info("Built HRRR -> NE-grid regrid weights "
+                        "(one-time setup, ~0.3s)")
+
+        regridded = _regrid(field2d, _REGRID_CACHE["weights_pack"])
+        channels.append(regridded.astype(np.float32))
+
+    tensor = np.stack(channels, axis=-1)
+    if np.isnan(tensor).any():
+        raise RuntimeError(
+            f"NaN in regridded HRRR tensor for "
+            f"{cycle_dt.isoformat()} f{fxx:02d}")
+    return tensor
+
+
+def _fetch_with_cache(cycle_dt: datetime, fxx: int) -> np.ndarray:
+    """Fetch one (cycle, fxx) pair via cache or network."""
+    p = _cache_path(cycle_dt, fxx)
+    if p.exists():
+        try:
+            with np.load(p) as f:
+                return f["weather"].astype(np.float32)
+        except Exception:  # corrupt cache file, refetch
+            p.unlink(missing_ok=True)
+    tensor = _fetch_one_via_herbie(cycle_dt, fxx)
+    # Store as float16 to halve disk usage (~2.8 MB/file vs 5.6 MB)
+    np.savez_compressed(p, weather=tensor.astype(np.float16))
+    return tensor
+
+
+def _fetch_parallel(jobs: Sequence[tuple[datetime, int]],
+                     parallel: int = 8,
+                     progress: Optional[Callable[[int, int, str], None]] = None,
+                     ) -> dict[tuple[datetime, int], np.ndarray]:
+    """Fetch many (cycle_dt, fxx) pairs in parallel; return dict by job key."""
+    if not jobs:
+        return {}
+    out: dict[tuple[datetime, int], np.ndarray] = {}
+    if parallel <= 1:
+        for i, (cdt, fxx) in enumerate(jobs):
+            out[(cdt, fxx)] = _fetch_with_cache(cdt, fxx)
+            if progress:
+                progress(i + 1, len(jobs), f"{cdt.strftime('%Y-%m-%d %H')} f{fxx:02d}")
+        return out
+
+    with ThreadPoolExecutor(max_workers=parallel) as ex:
+        futures = {ex.submit(_fetch_with_cache, cdt, fxx): (cdt, fxx)
+                   for cdt, fxx in jobs}
+        completed = 0
+        for fut in as_completed(futures):
+            key = futures[fut]
+            out[key] = fut.result()
+            completed += 1
+            if progress:
+                cdt, fxx = key
+                progress(completed, len(jobs),
+                         f"{cdt.strftime('%Y-%m-%d %H')} f{fxx:02d}")
+    return out
+
+
+# =====================================================================
+#  Public API
+# =====================================================================
+
+def fetch_history(end_dt: datetime, hours: int = 24,
+                   parallel: int = 8,
+                   progress: Optional[Callable[[int, int, str], None]] = None,
+                   ) -> np.ndarray:
+    """Return ``(hours, 450, 449, 7)`` float32 of HRRR f00 analyses for
+    the inclusive window ``[end_dt - hours, end_dt - 1h]``.
+
+    Each requested valid-hour ``H`` uses cycle ``H`` with fxx=0 (i.e.,
+    the analysis at that valid hour), matching how the training data
+    was constructed.
+    """
+    end_dt = _hour_floor_utc(end_dt)
+    valid_hours = [end_dt - timedelta(hours=hours - i) for i in range(hours)]
+    jobs = [(vh, 0) for vh in valid_hours]
+    fetched = _fetch_parallel(jobs, parallel=parallel, progress=progress)
+    out = np.stack([fetched[(vh, 0)] for vh in valid_hours], axis=0)
+    return out
+
+
+# HRRR cycles with extended (0-48 h) forecasts. Other hourly cycles
+# (01/02/04/05/...) only go out to f18, so we can't get 24 h from them.
+LONG_CYCLE_HOURS = (0, 6, 12, 18)
+
+
+def _latest_long_cycle_le(dt: datetime) -> datetime:
+    """Return the most recent HRRR long cycle (00/06/12/18 UTC) <= dt."""
+    dt = _hour_floor_utc(dt)
+    while dt.hour not in LONG_CYCLE_HOURS:
+        dt -= timedelta(hours=1)
+    return dt
+
+
+def fetch_forecast_for_window(target_start: datetime, hours: int = 24,
+                                publication_lag_hours: int = 2,
+                                parallel: int = 8,
+                                progress: Optional[Callable[[int, int, str], None]] = None,
+                                ) -> tuple[np.ndarray, datetime, int]:
+    """Return ``(hours, 450, 449, 7)`` covering valid hours
+    ``[target_start, target_start + hours - 1]``, using the most recent
+    HRRR long cycle (one of 00/06/12/18 UTC) that was published before
+    ``target_start`` (with ``publication_lag_hours`` margin to allow for
+    cycle processing delay).
+
+    Returns ``(weather, cycle_dt, fxx_start)`` so the caller can log
+    which cycle was used.
+    """
+    target_start = _hour_floor_utc(target_start)
+    cutoff = target_start - timedelta(hours=publication_lag_hours)
+    cycle_dt = _latest_long_cycle_le(cutoff)
+    fxx_start = int((target_start - cycle_dt).total_seconds() / 3600)
+    jobs = [(cycle_dt, fxx) for fxx in range(fxx_start, fxx_start + hours)]
+    fetched = _fetch_parallel(jobs, parallel=parallel, progress=progress)
+    out = np.stack([fetched[(cycle_dt, fxx)]
+                    for fxx in range(fxx_start, fxx_start + hours)], axis=0)
+    return out, cycle_dt, fxx_start
+
+
+def fetch_forecast(cycle_dt: datetime, hours: int = 24,
+                    parallel: int = 8,
+                    progress: Optional[Callable[[int, int, str], None]] = None,
+                    ) -> np.ndarray:
+    """Backwards-compat wrapper: fetch f01..f{hours} from a specific cycle.
+
+    NOTE: only long cycles (00/06/12/18 UTC) reliably cover 24+ hours.
+    For automatic cycle selection, prefer ``fetch_forecast_for_window``.
+    """
+    cycle_dt = _hour_floor_utc(cycle_dt)
+    jobs = [(cycle_dt, fxx) for fxx in range(1, hours + 1)]
+    fetched = _fetch_parallel(jobs, parallel=parallel, progress=progress)
+    out = np.stack([fetched[(cycle_dt, fxx)] for fxx in range(1, hours + 1)],
+                   axis=0)
+    return out
+
+
+def latest_available_cycle(target_dt: datetime,
+                             max_lookback_hours: int = 4,
+                             ) -> datetime:
+    """Find the most recent HRRR cycle <= ``target_dt`` whose forecast
+    hours appear to be on S3 (HRRR has ~1-2 hour publication lag).
+
+    We probe by trying to instantiate Herbie for each cycle from
+    ``target_dt`` backwards, succeeding when ``H.grib`` resolves.
+    Returns the cycle datetime (UTC, hour-floored, naive).
+    """
+    target_dt = _hour_floor_utc(target_dt)
+    try:
+        from herbie import Herbie       # noqa: WPS433
+    except ImportError as e:
+        raise RuntimeError(f"herbie-data not installed: {e}") from e
+
+    for back in range(0, max_lookback_hours + 1):
+        cdt = target_dt - timedelta(hours=back)
+        try:
+            H = Herbie(cdt.strftime("%Y-%m-%d %H:00"),
+                       model="hrrr", product="sfc", fxx=1, verbose=False)
+            if H.grib is not None:
+                return cdt
+        except Exception:  # noqa: BLE001
+            continue
+    raise RuntimeError(
+        f"No HRRR cycle available within last {max_lookback_hours}h of "
+        f"{target_dt.isoformat()}")
+
+
+if __name__ == "__main__":
+    # Smoke test: fetch one f00 + one f01 from yesterday's noon cycle
+    logging.basicConfig(level=logging.INFO, format="%(message)s")
+    yesterday_noon = (datetime.now(timezone.utc) - timedelta(days=1)
+                       ).replace(hour=12, minute=0, second=0, microsecond=0,
+                                  tzinfo=None)
+
+    print(f"Smoke test cycle: {yesterday_noon} UTC")
+    arr = _fetch_with_cache(yesterday_noon, 0)
+    print(f"  f00: shape={arr.shape}, dtype={arr.dtype}, "
+          f"mean per channel: " + ", ".join(
+              f"{name}={arr[..., i].mean():.2f}" for i, (name, _) in enumerate(_CHANNELS)))
+    arr1 = _fetch_with_cache(yesterday_noon, 1)
+    print(f"  f01: shape={arr1.shape}, dtype={arr1.dtype}, "
+          f"mean per channel: " + ", ".join(
+              f"{name}={arr1[..., i].mean():.2f}" for i, (name, _) in enumerate(_CHANNELS)))
+    print(f"  cache dir: {CACHE_DIR}, n files: {len(list(CACHE_DIR.glob('*.npz')))}")

iso_ne_fetch.py

@@ -1,27 +1,21 @@
 """
-Fetch the past 24 hours of ISO-NE per-zone demand for the live demo.
+High-level ISO-NE per-zone demand fetcher for the Space.
 
-Three sources, in priority order:
+Wraps the low-level fetcher in ``iso_ne_zonal.py`` with:
 
-1. **EIA Open Data API** at https://api.eia.gov/v2/electricity/rto/region-data
-   (system-level hourly demand, respondent=ISNE).  Free, requires a
-   personal API key registered via https://www.eia.gov/opendata/register.php
-   and exposed to the Space as the secret `EIA_API_KEY`.  We split the
-   system total into the 8 ISO-NE zones using a fixed proportion
-   vector estimated from 2022 zonal load reports.
+  - In-memory cache (5-minute TTL) so repeated clicks within a few
+    minutes don't refetch from ISO-NE
+  - Optional bundled CSV fallback for offline / API-down scenarios
+  - Optional integration with a long-history CSV pulled from the data
+    repo at Space startup (used to seed Chronos context without
+    re-fetching 30 days of ISO-NE on every click)
 
-2. **ISO-NE legacy `wsclient` endpoint**.  Tried as a backup; in
-   practice it currently returns HTTP 500 from outside the IETF
-   network, so it almost always falls through.
+Public API kept stable so ``app.py`` can swap from the old EIA-based
+implementation without further changes:
 
-3. **Bundled CSV fallback** at `assets/sample_demand_2022.csv` (24 h)
-   and `assets/sample_demand_2022_long.csv` (720 h).  Used when both
-   live paths fail (no key configured, network down, rate-limited).
-
-True per-zone real-time data requires an authenticated ISO Express
-account. The proportional split is a reasonable demo approximation:
-the model sees real recent ISO-NE-wide demand patterns; only the
-per-zone allocation is fixed.
+  - ``ZONE_COLS``                          : list of 8 zone names
+  - ``fetch_recent_demand_mwh(end_dt)``    : (24, 8) MWh + source label
+  - ``fetch_long_history_mwh(end_dt, hours=720)`` : (hours, 8) MWh + label
 """
 
 from __future__ import annotations
@@ -29,6 +23,7 @@ from __future__ import annotations
 import logging
 import os
 from datetime import datetime, timedelta, timezone
+from io import StringIO
 from pathlib import Path
 from typing import Optional
 
@@ -36,214 +31,165 @@ import numpy as np
 import pandas as pd
 import requests
 
-ZONE_COLS = ["ME", "NH", "VT", "CT", "RI", "SEMA", "WCMA", "NEMA_BOST"]
-
-# Approximate zonal proportions of total ISO-NE demand,
-# derived from 2022 historical zonal load reports.
-# Sum is 1.0; values reflect typical share by zone.
-ZONE_PROPORTIONS = np.array([
-    0.064,  # ME
-    0.080,  # NH
-    0.045,  # VT
-    0.205,  # CT
-    0.070,  # RI
-    0.130,  # SEMA
-    0.115,  # WCMA
-    0.291,  # NEMA_BOST  (largest --- Boston metro)
-], dtype=np.float32)
-assert abs(ZONE_PROPORTIONS.sum() - 1.0) < 1e-3
+from iso_ne_zonal import ZONE_COLS, fetch_range, fetch_recent_hours
+
+logger = logging.getLogger(__name__)
 
 ASSETS_DIR = Path(__file__).parent / "assets"
 SAMPLE_CSV = ASSETS_DIR / "sample_demand_2022.csv"
-SAMPLE_CSV_LONG = ASSETS_DIR / "sample_demand_2022_long.csv"   # 720 h, 2022-12-02..12-31
+SAMPLE_CSV_LONG = ASSETS_DIR / "sample_demand_2022_long.csv"
 
-# In-memory cache: {timestamp_hash: (timestamp, ndarray)}
+# In-memory cache: { ("recent", end_hour) | ("long", end_hour, hours) -> (ts, np.ndarray) }
 _CACHE: dict = {}
-_CACHE_TTL_SECONDS = 300  # 5 minutes
-
-logger = logging.getLogger(__name__)
+_CACHE_TTL_SECONDS = 300
 
+# Path of the data-repo 30-day CSV (refreshed daily by GitHub Actions in
+# new-england-real-time-power-predict-data; downloaded by app.py at
+# startup and saved to /tmp). When present, fetch_long_history_mwh
+# uses it as the base and splices in the last 1-2 days from live API.
+DATA_REPO_30D_CSV_PATH = Path(os.environ.get(
+    "DATA_REPO_30D_CSV_PATH", "/tmp/iso_ne_30d.csv"))
 
-def _cache_key(end_dt: datetime) -> str:
-    return end_dt.strftime("%Y-%m-%dT%H:00")
 
+def _hour_floor_utc(dt: datetime) -> datetime:
+    if dt.tzinfo is None:
+        dt = dt.replace(tzinfo=timezone.utc)
+    return dt.astimezone(timezone.utc).replace(
+        minute=0, second=0, microsecond=0, tzinfo=None)
 
-EIA_API_URL = "https://api.eia.gov/v2/electricity/rto/region-data/data/"
 
-
-def _try_eia_api(end_dt: datetime, hours: int = 24) -> Optional[np.ndarray]:
-    """Fetch ISO-NE system demand from EIA Open Data.
-
-    Requires the env var ``EIA_API_KEY`` (registered free at
-    https://www.eia.gov/opendata/register.php and exposed to this
-    Space as a Secret).
-
-    Returns ``(hours, 8)`` MWh array on success, ``None`` on any failure
-    (no key, HTTP error, missing rows, parse error).
-    """
-    key = os.environ.get("EIA_API_KEY", "").strip()
-    if not key:
-        return None
-    try:
-        # EIA returns data on hour-ending convention; pull a generous
-        # window so we can clip the freshest `hours` hours.
-        start = (end_dt - timedelta(hours=hours + 6)).strftime("%Y-%m-%dT%H")
-        end = end_dt.strftime("%Y-%m-%dT%H")
-        params = {
-            "api_key": key,
-            "frequency": "hourly",
-            "data[0]": "value",
-            "facets[respondent][]": "ISNE",
-            "facets[type][]": "D",        # 'D' = demand
-            "start": start,
-            "end": end,
-            "sort[0][column]": "period",
-            "sort[0][direction]": "desc",
-            "length": hours + 24,
-        }
-        r = requests.get(EIA_API_URL, params=params, timeout=8)
-        if r.status_code != 200:
-            logger.info("EIA API HTTP %d: %s", r.status_code, r.text[:200])
-            return None
-        payload = r.json()
-        rows = payload.get("response", {}).get("data", [])
-        if not rows:
-            return None
-        df = pd.DataFrame(rows)
-        if "period" not in df.columns or "value" not in df.columns:
-            return None
-        df["ts"] = pd.to_datetime(df["period"], utc=True, errors="coerce")
-        df = df.dropna(subset=["ts"]).sort_values("ts")
-        df["value"] = pd.to_numeric(df["value"], errors="coerce")
-        df = df.dropna(subset=["value"])
-        if len(df) < hours:
-            return None
-        last = df.tail(hours)["value"].to_numpy(dtype=np.float32)
-        return _split_to_zones(last)
-    except Exception as e:  # noqa: BLE001
-        logger.info("EIA API fetch failed: %s", e)
-        return None
-
-
-def _try_iso_ne_api(end_dt: datetime) -> Optional[np.ndarray]:
-    """Backup: ISO-NE legacy wsclient endpoint.
-
-    Frequently returns HTTP 500 from outside their network, so this
-    is mostly a fallback after EIA. Returns ``(24, 8)`` MWh or ``None``.
-    """
-    try:
-        url = "https://www.iso-ne.com/ws/wsclient"
-        params = {
-            "_nstmp_formDate": int(end_dt.timestamp() * 1000),
-            "_nstmp_startDate": (end_dt - timedelta(hours=25)).strftime("%m/%d/%Y"),
-            "_nstmp_endDate":   end_dt.strftime("%m/%d/%Y"),
-            "_nstmp_chartName": "fuelmix",
-        }
-        r = requests.get(url, params=params, timeout=4)
-        if r.status_code != 200:
-            return None
-        data = r.json()
-        if not isinstance(data, list) or not data:
-            return None
-        df = pd.DataFrame(data)
-        if "BeginDate" not in df.columns or "GenMw" not in df.columns:
-            return None
-        df["ts"] = pd.to_datetime(df["BeginDate"])
-        hourly = df.groupby(df["ts"].dt.floor("h"))["GenMw"].sum().sort_index()
-        last24 = hourly.tail(24).values.astype(np.float32)
-        if len(last24) < 24:
-            return None
-        return _split_to_zones(last24)
-    except Exception as e:  # noqa: BLE001
-        logger.info("ISO-NE API fetch failed: %s", e)
+def _cache_get(key: tuple) -> Optional[np.ndarray]:
+    cached = _CACHE.get(key)
+    if cached is None:
         return None
+    ts, arr = cached
+    if (datetime.now(timezone.utc) - ts).total_seconds() < _CACHE_TTL_SECONDS:
+        return arr.copy()
+    return None
 
 
-def _split_to_zones(system_total: np.ndarray) -> np.ndarray:
-    """system_total: (24,) -> (24, 8) using ZONE_PROPORTIONS."""
-    return np.outer(system_total, ZONE_PROPORTIONS).astype(np.float32)
+def _cache_put(key: tuple, arr: np.ndarray) -> None:
+    _CACHE[(key)] = (datetime.now(timezone.utc), arr.copy())
 
 
-def _load_sample_csv() -> np.ndarray:
-    """Fallback: read 24-hour slice from bundled CSV."""
+def _load_sample_recent() -> np.ndarray:
     df = pd.read_csv(SAMPLE_CSV)
     arr = df[ZONE_COLS].tail(24).to_numpy(dtype=np.float32)
     if arr.shape != (24, 8):
-        raise RuntimeError(f"Sample CSV has wrong shape {arr.shape}, expected (24, 8)")
+        raise RuntimeError(
+            f"Bundled sample_demand_2022.csv has wrong shape {arr.shape}")
     return arr
 
 
-def fetch_recent_demand_mwh(end_dt: Optional[datetime] = None):
-    """Fetch (24, 8) MWh array for the 24h ending at end_dt.
+def _load_sample_long(hours: int) -> np.ndarray:
+    if SAMPLE_CSV_LONG.exists():
+        df = pd.read_csv(SAMPLE_CSV_LONG)
+        arr = df[ZONE_COLS].tail(hours).to_numpy(dtype=np.float32)
+        if arr.shape == (hours, 8):
+            return arr
+    short = _load_sample_recent()
+    return np.tile(short, (hours // 24 + 1, 1))[:hours].astype(np.float32)
+
 
-    Returns (array, source_label) where source_label is "live" if the
-    API succeeded, "cached" if we used the in-memory cache, or
-    "sample-2022" if we fell back to the bundled CSV.
+def fetch_recent_demand_mwh(end_dt: Optional[datetime] = None
+                              ) -> tuple[np.ndarray, str]:
+    """Return ``(24, 8)`` MWh for the most recent 24 contiguous hours
+    ending at ``end_dt`` (or now). Source label is one of:
+      - ``"live (ISO-NE 5-min zonal -> hourly)"``
+      - ``"cached"``
+      - ``"sample-2022"``
     """
     if end_dt is None:
-        end_dt = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
-
-    key = _cache_key(end_dt)
-    cached = _CACHE.get(key)
+        end_dt = datetime.now(timezone.utc)
+    end_dt = _hour_floor_utc(end_dt)
+    cache_key = ("recent", end_dt)
+    cached = _cache_get(cache_key)
     if cached is not None:
-        ts, arr = cached
-        if (datetime.now(timezone.utc) - ts).total_seconds() < _CACHE_TTL_SECONDS:
-            return arr.copy(), "cached"
-
-    arr = _try_eia_api(end_dt, hours=24)
-    if arr is not None:
-        _CACHE[key] = (datetime.now(timezone.utc), arr)
-        return arr.copy(), "live (EIA)"
+        return cached, "cached"
+    try:
+        arr, latest = fetch_recent_hours(end_dt, hours=24)
+        _cache_put(cache_key, arr)
+        lag_hours = (end_dt - latest).total_seconds() / 3600
+        label = f"live (ISO-NE 5-min zonal, latest hour {latest.isoformat()}, "
+        label += f"lag {lag_hours:.0f}h)" if lag_hours > 0 else f"live (ISO-NE 5-min zonal)"
+        return arr, label
+    except Exception as e:  # noqa: BLE001
+        logger.warning("ISO-NE realtime fetch failed: %s; falling back to bundled CSV", e)
+        return _load_sample_recent(), "sample-2022 (ISO-NE unreachable)"
 
-    arr = _try_iso_ne_api(end_dt)
-    if arr is not None:
-        _CACHE[key] = (datetime.now(timezone.utc), arr)
-        return arr.copy(), "live (ISO-NE)"
 
-    arr = _load_sample_csv()
-    return arr, "sample-2022"
+def _load_30d_base() -> Optional[pd.DataFrame]:
+    """Load data-repo's pre-built 30-day per-zone CSV if available."""
+    if not DATA_REPO_30D_CSV_PATH.exists():
+        return None
+    try:
+        df = pd.read_csv(DATA_REPO_30D_CSV_PATH, parse_dates=["timestamp_utc"])
+        df = df.set_index("timestamp_utc").sort_index()
+        return df[ZONE_COLS]
+    except Exception as e:  # noqa: BLE001
+        logger.warning("Failed to load 30d base CSV at %s: %s",
+                        DATA_REPO_30D_CSV_PATH, e)
+        return None
 
 
 def fetch_long_history_mwh(end_dt: Optional[datetime] = None,
-                            hours: int = 720):
-    """Fetch a long per-zone demand history (default 720 h = 30 days) ending
-    at end_dt, for use as Chronos-Bolt context.
+                             hours: int = 720
+                             ) -> tuple[np.ndarray, str]:
+    """Return ``(hours, 8)`` MWh of per-zone history ending at ``end_dt - 1h``.
 
     Strategy:
-      1. Read the bundled long-history CSV (720 hourly rows from 2022-12).
-      2. Splice in the 24 freshest hours from the live API / cache (so the
-         tail of the history reflects recent live demand) when available.
-
-    Returns:
-      (array of shape (hours, 8), source_label).  source_label ends in
-      "+live" when the tail 24 h came from the API, "+sample" otherwise.
+      1. If the data repo's 30d base CSV is present, start from it.
+      2. Otherwise fall back to the bundled long-history CSV.
+      3. Always splice the last ~24-48 hours from the live ISO-NE API
+         so the tail is fresh.
     """
     if end_dt is None:
-        end_dt = datetime.now(timezone.utc).replace(minute=0, second=0, microsecond=0)
-
-    # 1. Bundled long-history CSV (always present)
-    if not SAMPLE_CSV_LONG.exists():
-        # Fall back to short CSV repeated; less faithful but never crashes.
-        short = _load_sample_csv()
-        long_arr = np.tile(short, (hours // 24 + 1, 1))[:hours]
-        return long_arr.astype(np.float32), "sample-2022-tiled"
-
-    df = pd.read_csv(SAMPLE_CSV_LONG)
-    long_arr = df[ZONE_COLS].tail(hours).to_numpy(dtype=np.float32)
-    if long_arr.shape != (hours, 8):
-        # Something odd; return what we have and tag.
-        return long_arr.astype(np.float32), "sample-2022-short"
-
-    # 2. Try to splice 24 freshest hours from the live API
-    fresh = _try_iso_ne_api(end_dt)
-    if fresh is not None and fresh.shape == (24, 8):
-        long_arr[-24:] = fresh
-        return long_arr, "sample-2022+live"
-    return long_arr, "sample-2022"
-
-
-if __name__ == "__main__":
-    arr, src = fetch_recent_demand_mwh()
-    print(f"recent (24 h): source={src}, shape={arr.shape}")
-    long_arr, long_src = fetch_long_history_mwh()
-    print(f"long ({len(long_arr)} h): source={long_src}, shape={long_arr.shape}")
+        end_dt = datetime.now(timezone.utc)
+    end_dt = _hour_floor_utc(end_dt)
+    cache_key = ("long", end_dt, hours)
+    cached = _cache_get(cache_key)
+    if cached is not None:
+        return cached, "cached"
+
+    target_end = end_dt - timedelta(hours=1)        # last hour we want
+    target_start = target_end - timedelta(hours=hours - 1)
+
+    base = _load_30d_base()
+    base_label = "data-repo 30d"
+
+    if base is None:
+        long_arr = _load_sample_long(hours)
+        out = long_arr
+        _cache_put(cache_key, out)
+        return out, "sample-2022 (no data-repo CSV)"
+
+    # Try to splice live ISO-NE for the last 2 days for freshness
+    splice_label = ""
+    try:
+        live = fetch_range(target_end - timedelta(days=2), target_end,
+                            hourly=True)
+        # Overwrite overlapping rows in `base` with `live`
+        base.update(live)
+        splice_label = " + live splice"
+    except Exception as e:  # noqa: BLE001
+        logger.info("Live splice into long history failed: %s", e)
+
+    # Ensure we have continuous coverage; if base doesn't reach target_start,
+    # fall back to bundled long CSV for the missing tail
+    if base.index.min() > target_start:
+        logger.info("30d base starts at %s, missing %s -> %s; padding from sample",
+                     base.index.min(), target_start, base.index.min())
+        sample_long = _load_sample_long(hours)
+        out = sample_long
+    else:
+        # Slice exact window
+        idx = pd.date_range(start=target_start, end=target_end, freq="1h")
+        sliced = base.reindex(idx)
+        if sliced.isna().any().any():
+            logger.info("30d base has %d NaN rows in window; interpolating",
+                         int(sliced.isna().any(axis=1).sum()))
+            sliced = sliced.interpolate(method="time", limit=12).ffill().bfill()
+        out = sliced[ZONE_COLS].to_numpy(dtype=np.float32)
+
+    _cache_put(cache_key, out)
+    return out, base_label + splice_label

iso_ne_zonal.py

@@ -0,0 +1,239 @@
+"""
+Real-time ISO-NE per-zone demand fetcher (no auth required).
+
+Endpoint: https://www.iso-ne.com/transform/csv/fiveminuteestimatedzonalload
+Returns 5-minute estimated load for all 8 ISO-NE zones; we roll up to
+hourly (mean of 12 5-min observations) to match the model's input format.
+
+Required trick: the endpoint returns HTTP 403 to direct curl, but accepts
+the request once a session has visited a normal page first (cookie-prime
+pattern, borrowed from the gridstatus.io library at
+gridstatus/isone.py:_make_request).
+
+Zone IDs (ISO-NE locational tags) -> our column names:
+    4001 .Z.MAINE         -> ME
+    4002 .Z.NEWHAMPSHIRE  -> NH
+    4003 .Z.VERMONT       -> VT
+    4004 .Z.CONNECTICUT   -> CT
+    4005 .Z.RHODEISLAND   -> RI
+    4006 .Z.SEMASS        -> SEMA
+    4007 .Z.WCMASS        -> WCMA
+    4008 .Z.NEMASSBOST    -> NEMA_BOST
+
+Data publication delay is roughly 1 day: at 19:31 EDT today the CSV for
+yesterday is fully populated; intra-day data may be missing recent hours
+near the wall-clock present. The fetcher always asks for whole UTC days
+and the caller is responsible for trimming to the desired range.
+"""
+
+from __future__ import annotations
+
+import csv
+import io
+import logging
+from datetime import datetime, timedelta, timezone
+from typing import Optional
+
+import numpy as np
+import pandas as pd
+import requests
+
+ZONE_COLS = ["ME", "NH", "VT", "CT", "RI", "SEMA", "WCMA", "NEMA_BOST"]
+
+# ISO-NE locational identifiers (zone IDs in the public CSV)
+_ZONE_ID_TO_COL = {
+    4001: "ME",
+    4002: "NH",
+    4003: "VT",
+    4004: "CT",
+    4005: "RI",
+    4006: "SEMA",
+    4007: "WCMA",
+    4008: "NEMA_BOST",
+}
+
+_ZONAL_URL = "https://www.iso-ne.com/transform/csv/fiveminuteestimatedzonalload"
+_PRIME_URL = "https://www.iso-ne.com/isoexpress/web/reports/operations/-/tree/gen-fuel-mix"
+
+logger = logging.getLogger(__name__)
+
+
+def _new_session() -> requests.Session:
+    """Return a requests.Session that has cookies primed for ISO-NE."""
+    s = requests.Session()
+    s.headers.update({
+        "User-Agent": "Mozilla/5.0 (compatible; predict-power/1.0; "
+                      "https://github.com/jeffliulab/real-time-power-predict)",
+    })
+    s.get(_PRIME_URL, timeout=10)
+    return s
+
+
+def _parse_csv(text: str) -> pd.DataFrame:
+    """Parse ISO-NE's quoted-CSV format (rows prefixed with C/H/D markers).
+
+    Returns a DataFrame with columns: timestamp_utc, zone_id, zone_name,
+    native_load_mw, btm_solar_mw.
+    """
+    data_rows = [line for line in text.splitlines() if line.startswith('"D"')]
+    if not data_rows:
+        raise RuntimeError("ISO-NE CSV had no data rows")
+    parsed = list(csv.reader(data_rows, quotechar='"'))
+    df = pd.DataFrame(parsed, columns=[
+        "row_type", "datetime", "zone_id", "zone_name",
+        "native_load_mw", "btm_solar_mw",
+    ])
+    # ISO-NE timestamps in the CSV are local time without TZ marker but
+    # are documented as Eastern Prevailing Time. Localize then convert.
+    ts_local = pd.to_datetime(df["datetime"]).dt.tz_localize(
+        "US/Eastern", nonexistent="shift_forward", ambiguous="infer",
+    )
+    df["timestamp_utc"] = ts_local.dt.tz_convert("UTC").dt.tz_localize(None)
+    df["zone_id"] = df["zone_id"].astype(int)
+    df["native_load_mw"] = df["native_load_mw"].astype(float)
+    return df[["timestamp_utc", "zone_id", "zone_name", "native_load_mw"]]
+
+
+def fetch_one_day(date: datetime, session: Optional[requests.Session] = None,
+                   timeout: int = 20) -> pd.DataFrame:
+    """Fetch one calendar day of 5-minute per-zone estimated load.
+
+    Args:
+        date: any datetime; only the date portion (Eastern local) is used.
+        session: optional pre-primed session for batched fetches.
+
+    Returns:
+        Wide DataFrame indexed by timestamp_utc with one column per zone
+        (ME, NH, ..., NEMA_BOST), values in MWh-equivalent (5-min average MW
+        which when multiplied by 5/60 hours equals MWh; we keep MW units
+        and aggregate to hourly mean which numerically equals hourly MWh).
+    """
+    own_session = session is None
+    if own_session:
+        session = _new_session()
+    date_str = date.strftime("%Y%m%d")
+    url = f"{_ZONAL_URL}?start={date_str}&end={date_str}"
+    r = session.get(url, timeout=timeout)
+    if r.status_code != 200:
+        raise RuntimeError(
+            f"ISO-NE zonal fetch failed: HTTP {r.status_code} for {url}")
+    if "text/csv" not in r.headers.get("Content-Type", "").lower():
+        raise RuntimeError(
+            f"ISO-NE zonal fetch returned non-CSV: {r.headers.get('Content-Type')}")
+    long_df = _parse_csv(r.content.decode("utf8"))
+    long_df["zone"] = long_df["zone_id"].map(_ZONE_ID_TO_COL)
+    if long_df["zone"].isna().any():
+        unknown = long_df.loc[long_df["zone"].isna(), "zone_id"].unique().tolist()
+        raise RuntimeError(f"Unknown zone IDs in ISO-NE response: {unknown}")
+    wide = long_df.pivot_table(
+        index="timestamp_utc", columns="zone", values="native_load_mw",
+        aggfunc="first")
+    wide = wide[ZONE_COLS]            # canonical column order
+    wide.index.name = "timestamp_utc"
+    return wide
+
+
+def fetch_range(start_date: datetime, end_date: datetime,
+                  hourly: bool = True) -> pd.DataFrame:
+    """Fetch 5-minute (or hourly-rolled) per-zone load over an inclusive
+    date range [start_date, end_date].
+
+    Args:
+        start_date / end_date: datetimes; only the date portion is used.
+            Both endpoints are inclusive.
+        hourly: if True (default), aggregate 12 5-min bins per hour to
+            the hourly mean (matches model input format). If False, return
+            the raw 5-minute resolution.
+
+    Returns:
+        DataFrame with timestamp_utc index and 8 zone columns.
+    """
+    if start_date.tzinfo is not None:
+        start_date = start_date.astimezone(timezone.utc).replace(tzinfo=None)
+    if end_date.tzinfo is not None:
+        end_date = end_date.astimezone(timezone.utc).replace(tzinfo=None)
+
+    session = _new_session()
+    parts = []
+    cur = start_date.replace(hour=0, minute=0, second=0, microsecond=0)
+    end = end_date.replace(hour=0, minute=0, second=0, microsecond=0)
+    while cur <= end:
+        try:
+            parts.append(fetch_one_day(cur, session=session))
+        except Exception as e:  # noqa: BLE001
+            logger.warning("ISO-NE fetch for %s failed: %s", cur.date(), e)
+        cur += timedelta(days=1)
+
+    if not parts:
+        raise RuntimeError(
+            f"ISO-NE fetch returned no data for range "
+            f"{start_date.date()} -> {end_date.date()}")
+
+    df = pd.concat(parts).sort_index()
+    df = df[~df.index.duplicated(keep="last")]
+    if not hourly:
+        return df
+
+    hourly_df = df.resample("1h").mean(numeric_only=True)
+    hourly_df = hourly_df[ZONE_COLS]
+    return hourly_df
+
+
+def fetch_recent_hours(end_dt: datetime, hours: int = 24,
+                        max_lookback_days: int = 3
+                        ) -> tuple[np.ndarray, datetime]:
+    """Return ``(hours, 8)`` MW array of the most recent complete hours.
+
+    ISO-NE 5-min zonal data has ~1-2 hour publication lag. This helper
+    looks back from ``end_dt`` (rounded down to the hour) and finds the
+    latest contiguous window of ``hours`` complete hours of per-zone data
+    among the last ``max_lookback_days`` UTC dates.
+
+    Returns:
+        (array of shape (hours, 8) float32, latest_timestamp_in_window).
+
+    Raises RuntimeError if there isn't a contiguous ``hours``-window in
+    the last ``max_lookback_days``.
+    """
+    if end_dt.tzinfo is None:
+        end_dt = end_dt.replace(tzinfo=timezone.utc)
+    end_dt = end_dt.astimezone(timezone.utc).replace(
+        minute=0, second=0, microsecond=0, tzinfo=None)
+
+    fetch_start = end_dt - timedelta(days=max_lookback_days)
+    df = fetch_range(fetch_start, end_dt, hourly=True)
+    df = df.dropna()                              # only fully-populated hours
+    if len(df) < hours:
+        raise RuntimeError(
+            f"ISO-NE has only {len(df)} complete hourly rows in the last "
+            f"{max_lookback_days} days; need {hours}.")
+
+    # Find the latest contiguous `hours`-length stretch (1-hour gaps allowed
+    # are NOT allowed here; we want strictly contiguous data).
+    df = df.sort_index()
+    contig_end = df.index[-1]
+    contig_start = contig_end - timedelta(hours=hours - 1)
+    window = df.loc[contig_start:contig_end]
+    if len(window) != hours:
+        raise RuntimeError(
+            f"ISO-NE: last {hours} hours not contiguous "
+            f"(got {len(window)} of {hours} expected, latest={contig_end}).")
+    return window[ZONE_COLS].to_numpy(dtype=np.float32), contig_end.to_pydatetime()
+
+
+if __name__ == "__main__":
+    # Smoke test
+    logging.basicConfig(level=logging.INFO, format="%(message)s")
+    yesterday = (datetime.now(timezone.utc) - timedelta(days=1))
+    print(f"Fetching one day of ISO-NE per-zone load for "
+          f"{yesterday.date()} (UTC)...")
+    df = fetch_one_day(yesterday)
+    print(f"  shape={df.shape}, columns={list(df.columns)}")
+    print(f"  first row: {df.iloc[0].to_dict()}")
+    print()
+    print("Fetching last 24 contiguous hours...")
+    arr, latest = fetch_recent_hours(datetime.now(timezone.utc), hours=24)
+    print(f"  shape={arr.shape}, latest_timestamp={latest}")
+    print(f"  sum_at_t0={arr.sum(axis=1)[0]:.0f} MW")
+    print(f"  zone means: "
+          + ", ".join(f"{z}={arr[:, i].mean():.0f}" for i, z in enumerate(ZONE_COLS)))

model_utils.py

@@ -68,11 +68,22 @@ def denormalize_demand(z: np.ndarray, norm_stats: dict) -> np.ndarray:
     return (z * std + mean).astype(np.float32)
 
 
+def normalize_weather(raster: np.ndarray, norm_stats: dict) -> np.ndarray:
+    """(T, H, W, 7) raw HRRR -> (T, H, W, 7) z-scored using training stats.
+
+    norm_stats stores per-channel mean/std as (1, 1, 1, 7) tensors.
+    """
+    mean = norm_stats["weather_mean"].cpu().numpy().reshape(1, 1, 1, -1)
+    std = norm_stats["weather_std"].cpu().numpy().reshape(1, 1, 1, -1)
+    return ((raster - mean) / std).astype(np.float32)
+
+
 def synthetic_weather_z(history_len: int = HISTORY_LEN,
                         future_len: int = FUTURE_LEN) -> np.ndarray:
     """Return a (S+24, H, W, C) array of zeros (training-mean weather
-    in z-score space). The baseline still produces calibrated per-zone
-    output because the tabular branch carries demand+calendar info."""
+    in z-score space). Kept as a fallback when the live HRRR fetcher
+    fails (e.g. no network, S3 outage); the model is degraded but still
+    produces calibrated output from demand + calendar."""
     return np.zeros((history_len + future_len, WEATHER_H, WEATHER_W, WEATHER_C),
                     dtype=np.float32)
 
@@ -83,20 +94,38 @@ def run_forecast(model: torch.nn.Module,
                  hist_cal: np.ndarray,
                  future_cal: np.ndarray,
                  norm_stats: dict,
+                 hist_weather_raw: np.ndarray,
+                 future_weather_raw: np.ndarray,
                  device: str = "cpu") -> np.ndarray:
-    """Run the baseline on synthetic weather + real demand history.
+    """Run the baseline forecast.
 
     Args:
-      hist_demand_mwh: (24, 8) recent ISO-NE per-zone demand in MWh.
-      hist_cal:        (24, 44) calendar features for the history window.
-      future_cal:      (24, 44) calendar features for the next 24 h.
+      hist_demand_mwh:    (24, 8) recent ISO-NE per-zone demand in MWh.
+      hist_cal:           (24, 44) calendar features for the history window.
+      future_cal:         (24, 44) calendar features for the next 24 h.
+      hist_weather_raw:   (24, 450, 449, 7) RAW HRRR f00 analyses for the
+                          history window. Will be z-scored internally.
+      future_weather_raw: (24, 450, 449, 7) RAW HRRR f01..f24 forecasts
+                          (or analyses, if available) for the future
+                          window. Will be z-scored internally.
 
     Returns:
       (24, 8) forecast in MWh.
     """
-    weather = synthetic_weather_z()                           # (48, H, W, C)
-    hist_w = torch.from_numpy(weather[:HISTORY_LEN]).unsqueeze(0).to(device)
-    fut_w = torch.from_numpy(weather[HISTORY_LEN:]).unsqueeze(0).to(device)
+    if hist_weather_raw.shape != (HISTORY_LEN, WEATHER_H, WEATHER_W, WEATHER_C):
+        raise ValueError(
+            f"hist_weather_raw shape {hist_weather_raw.shape} != "
+            f"({HISTORY_LEN}, {WEATHER_H}, {WEATHER_W}, {WEATHER_C})")
+    if future_weather_raw.shape != (FUTURE_LEN, WEATHER_H, WEATHER_W, WEATHER_C):
+        raise ValueError(
+            f"future_weather_raw shape {future_weather_raw.shape} != "
+            f"({FUTURE_LEN}, {WEATHER_H}, {WEATHER_W}, {WEATHER_C})")
+
+    hist_w_z = normalize_weather(hist_weather_raw, norm_stats)
+    fut_w_z = normalize_weather(future_weather_raw, norm_stats)
+
+    hist_w = torch.from_numpy(hist_w_z).unsqueeze(0).to(device)
+    fut_w = torch.from_numpy(fut_w_z).unsqueeze(0).to(device)
 
     hist_y_z = normalize_demand(hist_demand_mwh, norm_stats)
     hist_y = torch.from_numpy(hist_y_z).unsqueeze(0).to(device)

packages.txt

@@ -0,0 +1,2 @@
+libeccodes-dev
+libeccodes-tools

requirements.txt

@@ -2,9 +2,16 @@ gradio>=5.30,<6
 torch>=2.5,<3
 numpy>=1.26,<3
 pandas>=2.0
+scipy>=1.11
 plotly>=5.18
 requests>=2.31
-# Chronos-Bolt foundation model for the optional ensemble path.
-# Only required when the user selects the Ensemble model in the UI;
-# the Baseline-only path does not import it.
+
+# Real-time HRRR weather (fetched on-demand for live forecasts and as
+# the input to the rolling backtest cache that ships from the data repo).
+herbie-data>=2024.1
+cfgrib>=0.9.10
+xarray>=2024.0
+eccodes>=2.40
+
+# Chronos-Bolt foundation model for the per-zone ensemble path.
 chronos-forecasting>=1.5,<2