Fix HF deployment serving stale BESS list with fictional DUIDs

The HuggingFace Space was returning a hand-written fallback bess_list.json
on every /api/bess call because AEMO's WAF on www.aemo.com.au returns 403
to requests from the HF egress IP range (verified in the HF container
logs — UA-spoofing alone doesn't help, the block is IP-based). The
fallback had stale fictional DUIDs (NESBESS1, HORNSDALE_PWR1, etc.) and no
capacity_mwh field, which caused every BESS to show "? MWh" in the
dropdown and every SCADA/dispatch lookup to 404.

Five fixes shipped together:

1. Send proper browser headers on the AEMO XLSX request (low-confidence
shot — the block is IP-based, but no reason not to try).

2. Bake a fresh bess_list.json snapshot into the repo daily via a new
GitHub Action (.github/workflows/refresh_bess_list.yml) running
scripts/refresh_bess_list.py from GH runners (egress NOT on AEMO's
blocklist). The bundled snapshot now has 38 real BESS units with
real DUIDs and capacity_mwh values.

3. Mirror the raw XLSX through raw.githubusercontent.com. gen_info_fetcher
now tries live AEMO -> mirror -> bundled snapshot in order, so a
request only hits the snapshot when both network sources fail.

4. Surface provenance via /api/bess. New response shape:
{states, source, fetched_at, warnings}. Frontend displays an amber
banner when 'source' is not 'live'. BREAKING: callers reading
/api/bess as {state: [...]} directly must read .states.

5. Update DATA_START_DATE 2025-02-28 -> 2025-04-29 and
DISPATCH_START_DATE 2025-02-11 -> 2025-04-01 to reflect the earliest
bundles actually still hosted on NEMWEB. Earlier Era 1 archive
bundles have been rolled off. Error messages and About/README text
updated to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

.github/workflows/refresh_bess_list.yml

@@ -0,0 +1,44 @@
+name: Refresh BESS list snapshot
+
+on:
+  schedule:
+    # Daily at 02:00 UTC (~12:00 AEST). AEMO publishes Gen Info updates monthly,
+    # so most days produce no diff.
+    - cron: "0 2 * * *"
+  workflow_dispatch: {}
+
+permissions:
+  contents: write
+
+jobs:
+  refresh:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: master
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install minimal deps
+        run: pip install --no-cache-dir httpx==0.27.* openpyxl==3.1.*
+
+      - name: Refresh snapshot
+        run: python scripts/refresh_bess_list.py
+
+      - name: Commit if changed
+        run: |
+          set -e
+          # Only commit if the parsed JSON or raw XLSX changed.
+          # The meta file changes every run (timestamp); ignore it on its own.
+          if git diff --quiet -- app/data/bess_list.json app/data/aemo_gen_info.xlsx; then
+            echo "No change in bess_list.json or aemo_gen_info.xlsx; skipping commit."
+            exit 0
+          fi
+          git config user.name  "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add app/data/bess_list.json app/data/aemo_gen_info.xlsx app/data/bess_list_meta.json
+          git commit -m "Refresh BESS list snapshot from AEMO XLSX"
+          git push origin master

README.md

@@ -20,7 +20,7 @@ An interactive web app for exploring high-frequency power and energy storage dat
 ### 4-second SCADA — `PUBLIC_NEXT_DAY_FPPMW`
 
 - **Source:** [AEMO NEMWEB FPPDAILY](https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/)
-- **Available from:** 28 February 2025 (FPP scheme commencement)
+- **Available from:** 29 April 2025 (the FPP scheme itself commenced 28 February 2025, but earlier daily bundles have been rolled off the AEMO NEMWEB archive)
 - **Granularity:** one row per 4-second grid slot
 
 | Column | Description |
@@ -33,7 +33,7 @@ An interactive web app for exploring high-frequency power and energy storage dat
 ### 5-minute Dispatch — `DISPATCH_UNIT_SOLUTION`
 
 - **Source:** [AEMO NEMWEB Next_Day_Dispatch](https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/)
-- **Available from:** 11 February 2025
+- **Available from:** 1 April 2025 (the `INITIAL_ENERGY_STORAGE` column was added on 11 February 2025, but earlier monthly bundles have been rolled off the AEMO NEMWEB archive)
 - **Granularity:** one row per 5-minute dispatch interval
 
 | Column | Description |

app/config.py

@@ -5,8 +5,16 @@ from datetime import date
 AEMO_CURRENT_URL = "https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/"
 AEMO_ARCHIVE_URL = "https://nemweb.com.au/Reports/Archive/FPPDAILY/"
 
-# Data availability: first FPPMW SCADA bundle is PUBLIC_NEXT_DAY_FPPMW_20250228.zip
-DATA_START_DATE = date(2025, 2, 28)
+# Data availability: AEMO has rolled off the early Era 1 archive bundles.
+# The oldest archive bundle currently on NEMWEB is PUBLIC_NEXT_DAY_FPP_20250430.zip
+# whose first inner file is publication 30 Apr 2025 = settlement 29 Apr 2025
+# (the first day of Era 2 in our era model). FPPMW Format-1 bundles start the
+# same day. So 29 Apr 2025 is the earliest settlement date for which any
+# FPPDAILY data can still be fetched.
+#
+# The FPP scheme itself commenced on 28 Feb 2025, but the Era 1 monthly
+# bundles (Dec 2024 – 28 Apr 2025) have all been rolled off the archive.
+DATA_START_DATE = date(2025, 4, 29)
 
 # From this date onward, data is served as individual daily ZIPs from the
 # Current directory (faster).  Before this date, large archive bundles are
@@ -18,8 +26,11 @@ FPPMW_CUTOVER_DATE = date(2026, 1, 11)
 DISPATCH_CURRENT_URL = "https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/"
 DISPATCH_ARCHIVE_URL = "https://www.nemweb.com.au/REPORTS/ARCHIVE/Next_Day_Dispatch/"
 
-# Dispatch data available from 11 Feb 2025 (INITIAL_ENERGY_STORAGE column).
-DISPATCH_START_DATE = date(2025, 2, 11)
+# Dispatch data: the oldest archive bundle on NEMWEB is
+# PUBLIC_NEXT_DAY_DISPATCH_20250401.zip (covering settlement dates
+# 1 Apr 2025 – 30 Apr 2025). The INITIAL_ENERGY_STORAGE column has existed
+# since 11 Feb 2025, but earlier monthly bundles have been rolled off.
+DISPATCH_START_DATE = date(2025, 4, 1)
 
 # ── Request limits ────────────────────────────────────────────────────────────
 MAX_DAYS_PER_REQUEST = 1

app/data/bess_list.json

@@ -1,32 +1,277 @@
 {
   "NSW": [
-    {"duid": "NESBESS1", "name": "New England BESS 1", "capacity_mw": 126, "region": "NSW1"},
-    {"duid": "NESBESS2", "name": "New England BESS 2", "capacity_mw": 126, "region": "NSW1"},
-    {"duid": "WARATAHSF1", "name": "Waratah Super Battery 1", "capacity_mw": 850, "region": "NSW1"},
-    {"duid": "ERARINGSF1", "name": "Eraring Battery 1", "capacity_mw": 460, "region": "NSW1"},
-    {"duid": "LMOBESS1", "name": "Limondale BESS", "capacity_mw": 50, "region": "NSW1"}
-  ],
-  "VIC": [
-    {"duid": "BNGSF2", "name": "Big Battery (Victorian Big Battery)", "capacity_mw": 300, "region": "VIC1"},
-    {"duid": "RANGEBANKBESS1", "name": "Rangebank BESS", "capacity_mw": 200, "region": "VIC1"},
-    {"duid": "MREHABESS1", "name": "Melbourne Renewable Energy Hub BESS", "capacity_mw": 600, "region": "VIC1"},
-    {"duid": "GANNBESS1", "name": "Gannawarra BESS", "capacity_mw": 25, "region": "VIC1"},
-    {"duid": "BALLBESS1", "name": "Ballarat BESS", "capacity_mw": 30, "region": "VIC1"}
+    {
+      "capacity_mw": 10.0,
+      "capacity_mwh": 20.0,
+      "duid": "QBYNB1",
+      "name": "QBESS",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 60.28,
+      "capacity_mwh": 60.28,
+      "duid": "BHB1",
+      "name": "BKBESS1",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 99.99,
+      "capacity_mwh": 199.98,
+      "duid": "CAPBES1",
+      "name": "CBESS",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 24.96,
+      "capacity_mwh": 50.08,
+      "duid": "DPNTB1",
+      "name": "Darlington Pnt ESS1",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 460.0,
+      "capacity_mwh": 1073.0,
+      "duid": "ERB01",
+      "name": "ERARNGB1",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 60.0,
+      "capacity_mwh": 120.0,
+      "duid": "RESS1",
+      "name": "RiverESS",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 65.0,
+      "capacity_mwh": 130.0,
+      "duid": "RIVNB2",
+      "name": "Riverina ESS 2",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 86.4,
+      "capacity_mwh": 129.96,
+      "duid": "SMTHBES1",
+      "name": "SmithBes",
+      "region": "NSW1"
+    },
+    {
+      "capacity_mw": 61.2,
+      "capacity_mwh": 2700.0,
+      "duid": "WALGRV1",
+      "name": "WLGBESS1",
+      "region": "NSW1"
+    }
   ],
   "QLD": [
-    {"duid": "WDWBESS1", "name": "Western Downs BESS", "capacity_mw": 255, "region": "QLD1"},
-    {"duid": "GRNBKBESS1", "name": "Greenbank BESS", "capacity_mw": 200, "region": "QLD1"},
-    {"duid": "TRNBESS1", "name": "Tarong BESS", "capacity_mw": 300, "region": "QLD1"},
-    {"duid": "SWANBESS1", "name": "Swanbank BESS", "capacity_mw": 250, "region": "QLD1"},
-    {"duid": "WANDBESS1", "name": "Wandoan South BESS", "capacity_mw": 100, "region": "QLD1"}
+    {
+      "capacity_mw": 50.0,
+      "capacity_mwh": 100.0,
+      "duid": "BBATTERY1",
+      "name": "BLDRCMB1",
+      "region": "QLD1"
+    },
+    {
+      "capacity_mw": 100.0,
+      "capacity_mwh": 200.0,
+      "duid": "CHBESS1",
+      "name": "Chinch1",
+      "region": "QLD1"
+    },
+    {
+      "capacity_mw": 200.0,
+      "capacity_mwh": 400.0,
+      "duid": "GREENB1",
+      "name": "GREENB1",
+      "region": "QLD1"
+    },
+    {
+      "capacity_mw": 2.0,
+      "capacity_mwh": 4.0,
+      "duid": "KEPBG1",
+      "name": "KEPBG1",
+      "region": "QLD1"
+    },
+    {
+      "capacity_mw": 154.96,
+      "capacity_mwh": 297.96,
+      "duid": "ULPBESS1",
+      "name": "ULINDAB1",
+      "region": "QLD1"
+    },
+    {
+      "capacity_mw": 100.0,
+      "capacity_mwh": 150.0,
+      "duid": "WANDB1",
+      "name": "WSBESS1",
+      "region": "QLD1"
+    }
   ],
   "SA": [
-    {"duid": "HORNSDALE_PWR1", "name": "Hornsdale Power Reserve 1", "capacity_mw": 100, "region": "SA1"},
-    {"duid": "HORNSDALE_PWR2", "name": "Hornsdale Power Reserve 2", "capacity_mw": 100, "region": "SA1"},
-    {"duid": "HORNSDALE_PWR3", "name": "Hornsdale Power Reserve 3", "capacity_mw": 50, "region": "SA1"},
-    {"duid": "BLYTHBESS1", "name": "Blyth BESS", "capacity_mw": 200, "region": "SA1"},
-    {"duid": "DALRYMPLE1", "name": "Dalrymple BESS", "capacity_mw": 30, "region": "SA1"},
-    {"duid": "OSBORNE1", "name": "Osborne BESS", "capacity_mw": 50, "region": "SA1"}
+    {
+      "capacity_mw": 7.76,
+      "capacity_mwh": 12.6,
+      "duid": "ADPBA1",
+      "name": "ADPBA1",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 200.32,
+      "capacity_mwh": 400.0,
+      "duid": "BLYTHB1",
+      "name": "BLYTHWBT",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 3.08,
+      "capacity_mwh": 5.04,
+      "duid": "BOWWBA1",
+      "name": "BOWWBA1",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 2.16,
+      "capacity_mwh": null,
+      "duid": "CBWWBA1",
+      "name": "CBWWBA1",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 30.0,
+      "capacity_mwh": 9.0,
+      "duid": "DALNTH1",
+      "name": "BESS1-12",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 5.52,
+      "capacity_mwh": 8.82,
+      "duid": "HVWWBA1",
+      "name": "HVWWBA1",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 50.0,
+      "capacity_mwh": 61.3,
+      "duid": "HPR1",
+      "name": "HPRG EXP",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 100.0,
+      "capacity_mwh": 117.0,
+      "duid": "HPR1",
+      "name": "HPRG1",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 28.8,
+      "capacity_mwh": 51.84,
+      "duid": "LBB1",
+      "name": "LBB1",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 100.0,
+      "capacity_mwh": 200.0,
+      "duid": "MANNUMB1",
+      "name": "MANNUMBT",
+      "region": "SA1"
+    },
+    {
+      "capacity_mw": 250.7,
+      "capacity_mwh": 249.61,
+      "duid": "TIB1",
+      "name": "TIPSBESS",
+      "region": "SA1"
+    }
   ],
-  "TAS": []
+  "TAS": [],
+  "VIC": [
+    {
+      "capacity_mw": 30.09,
+      "capacity_mwh": 30.09,
+      "duid": "BALB1",
+      "name": "Ballarat Energy Storage System:1",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 20.0,
+      "capacity_mwh": 34.0,
+      "duid": "BULBES1",
+      "name": "Bulgana Green Power Hub - BESS:1",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 25.5,
+      "capacity_mwh": 50.0,
+      "duid": "GANNB1",
+      "name": "Gannawarra Energy Storage System:1",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 200.07,
+      "capacity_mwh": 162.0,
+      "duid": "HBESS1",
+      "name": "HBESS1",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 185.0,
+      "capacity_mwh": 370.0,
+      "duid": "KESSB1",
+      "name": "Koorangi",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 97.9,
+      "capacity_mwh": 195.8,
+      "duid": "LVES1",
+      "name": "LaTroBES",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 216.16,
+      "capacity_mwh": 399.84,
+      "duid": "MREHA1",
+      "name": "MREHA1",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 216.16,
+      "capacity_mwh": 399.84,
+      "duid": "MREHA2",
+      "name": "MREHA2",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 215.6,
+      "capacity_mwh": 800.8,
+      "duid": "MREHA3",
+      "name": "MREHA3",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 5.0,
+      "capacity_mwh": 10.0,
+      "duid": "PIBESS1",
+      "name": "PIBS_GU1",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 200.0,
+      "capacity_mwh": 400.0,
+      "duid": "RANGEB1",
+      "name": "Rangebnk",
+      "region": "VIC1"
+    },
+    {
+      "capacity_mw": 300.0,
+      "capacity_mwh": 470.0,
+      "duid": "VBB1",
+      "name": "BESS",
+      "region": "VIC1"
+    }
+  ]
 }

app/data/bess_list_meta.json

@@ -0,0 +1,6 @@
+{
+  "fetched_at": "2026-05-19T02:22:04+00:00",
+  "source_url": "https://www.aemo.com.au/-/media/files/electricity/nem/planning_and_forecasting/generation_information/2026/nem-generation-information-jan-2026.xlsx?rev=1f6bccf827284f9fb6d6f3ae56ed3fe9&sc_lang=en",
+  "xlsx_bytes": 871234,
+  "unit_count": 38
+}

app/routers/api.py

@@ -70,8 +70,24 @@ def _source_type(target_date: date) -> str:
 
 @router.get("/bess")
 async def get_bess_list():
-    """Return in-service battery storage units grouped by state."""
-    return await fetch_bess_list()
+    """
+    Return in-service battery storage units grouped by state.
+
+    Response shape:
+      {
+        "states":     { "NSW": [...], "VIC": [...], ... },
+        "source":     "live" | "mirror" | "snapshot",
+        "fetched_at": ISO-8601 timestamp,
+        "warnings":   list of strings (may be empty)
+      }
+    """
+    result = await fetch_bess_list()
+    return {
+        "states":     result.states,
+        "source":     result.source,
+        "fetched_at": result.fetched_at.isoformat(timespec="seconds") + "Z",
+        "warnings":   result.warnings,
+    }
 
 
 # ── Quality flags ─────────────────────────────────────────────────────────────

app/services/aemo_fetcher.py

@@ -492,8 +492,8 @@ async def fetch_csv_for_date(
     """
     if target_date < DATA_START_DATE:
         raise AEMOFetchError(
-            f"Data is only available from {DATA_START_DATE.strftime('%d %B %Y')} "
-            "when the FPP scheme commenced."
+            f"Data is only available from {DATA_START_DATE.strftime('%d %B %Y')}. "
+            "Earlier daily files have been rolled off the AEMO NEMWEB archive."
         )
     if not skip_future_check and target_date >= date.today():
         raise AEMOFetchError("Cannot request data for today or future dates.")

app/services/dispatch_fetcher.py

@@ -273,7 +273,8 @@ async def fetch_dispatch_csv_for_date(
     if target_date < DISPATCH_START_DATE:
         raise DispatchFetchError(
             f"Dispatch energy data is only available from "
-            f"{DISPATCH_START_DATE.strftime('%d %B %Y')}."
+            f"{DISPATCH_START_DATE.strftime('%d %B %Y')}. "
+            "Earlier monthly bundles have been rolled off the AEMO NEMWEB archive."
         )
     if target_date >= date.today():
         raise DispatchFetchError(

app/services/gen_info_fetcher.py

@@ -1,15 +1,23 @@
 """
 Fetches and parses the AEMO NEM Generation Information XLSX.
 
-Downloads the spreadsheet from AEMO, reads the "Generator Information" sheet,
-and returns all in-service battery storage units grouped by NEM region/state.
-
-The spreadsheet is cached in memory for 24 hours to avoid re-downloading on
-every request.  On any failure the caller receives the cached data (if any) or
-the static fallback bess_list.json.
+Sources tried in order on every cache miss:
+
+  1. live      — direct download from www.aemo.com.au (subject to WAF blocks
+                 from datacentre IPs; see _BROWSER_HEADERS).
+  2. mirror    — raw.githubusercontent.com copy committed daily by the
+                 refresh_bess_list workflow.
+  3. snapshot  — the parsed bess_list.json bundled in the repo (also
+                 written by refresh_bess_list).
+
+The first successful step is cached in memory for 24 hours.  The result
+carries a `source` field and any user-facing `warnings` so the API can
+expose data freshness in the response.
 """
 import io
+import json
 import logging
+from dataclasses import dataclass, field
 from datetime import datetime, timedelta
 from pathlib import Path
 
@@ -24,6 +32,38 @@ XLSX_URL = (
     "?rev=1f6bccf827284f9fb6d6f3ae56ed3fe9&sc_lang=en"
 )
 
+# raw.githubusercontent.com copy of the XLSX, refreshed daily by
+# .github/workflows/refresh_bess_list.yml. Used as the second-choice source
+# when the live AEMO fetch fails (e.g. WAF 403 from the HuggingFace egress IP).
+XLSX_MIRROR_URL = (
+    "https://raw.githubusercontent.com/pourmousavi/BESS-SCADA-Data"
+    "/master/app/data/aemo_gen_info.xlsx"
+)
+
+# AEMO's Azure Front Door WAF on www.aemo.com.au returns 403 to requests that
+# look like default Python clients. Sending a full browser-style header set
+# bypasses the simpler bot-detection rules. (Doesn't help against pure
+# IP-range blocks — see gen_info_mirror.py for the raw.githubusercontent.com
+# fallback.)
+_BROWSER_HEADERS = {
+    "User-Agent": (
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
+        "AppleWebKit/537.36 (KHTML, like Gecko) "
+        "Chrome/120.0.0.0 Safari/537.36"
+    ),
+    "Accept": (
+        "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet,"
+        "application/octet-stream;q=0.9,*/*;q=0.8"
+    ),
+    "Accept-Language": "en-AU,en;q=0.9",
+    "Accept-Encoding": "gzip, deflate, br",
+    "Referer": "https://www.aemo.com.au/energy-systems/electricity/national-electricity-market-nem/nem-forecasting-and-planning/forecasting-and-planning-data/generation-information",
+    "Sec-Fetch-Dest": "document",
+    "Sec-Fetch-Mode": "navigate",
+    "Sec-Fetch-Site": "same-origin",
+    "Upgrade-Insecure-Requests": "1",
+}
+
 SHEET_NAME = "Generator Information"
 
 # Map NEM region codes to the state keys used in the UI
@@ -49,8 +89,24 @@ _COL_CAPACITY_MWH = "Agg Nameplate Storage Capacity (MWh)"
 
 CACHE_TTL = timedelta(hours=24)
 
-# In-memory cache: (parsed_result, fetched_at)
-_cache: tuple[dict, datetime] | None = None
+
+@dataclass
+class BessListResult:
+    """Parsed BESS list plus provenance for the API response."""
+    states: dict[str, list[dict]]
+    source: str            # "live" | "mirror" | "snapshot"
+    fetched_at: datetime   # for the live/mirror path: now; for snapshot: snapshot timestamp
+    warnings: list[str] = field(default_factory=list)
+
+
+# In-memory cache: (result, cached_at)
+_cache: tuple[BessListResult, datetime] | None = None
+
+
+# Snapshot files written by scripts/refresh_bess_list.py
+_DATA_DIR = Path(__file__).parent.parent / "data"
+_SNAPSHOT_JSON = _DATA_DIR / "bess_list.json"
+_SNAPSHOT_META = _DATA_DIR / "bess_list_meta.json"
 
 
 def _find_col(headers: list[str], target: str) -> int | None:
@@ -166,12 +222,39 @@ def _parse_xlsx(xlsx_bytes: bytes) -> dict[str, list[dict]]:
     return result
 
 
-async def fetch_bess_list() -> dict[str, list[dict]]:
+async def _download_and_parse(url: str, client: httpx.AsyncClient) -> dict[str, list[dict]]:
+    resp = await client.get(url, headers=_BROWSER_HEADERS)
+    resp.raise_for_status()
+    return _parse_xlsx(resp.content)
+
+
+def _load_snapshot() -> BessListResult:
+    """Load the parsed JSON snapshot bundled in the repo. Raises if missing."""
+    states = json.loads(_SNAPSHOT_JSON.read_text())
+    fetched_at = datetime.utcnow()
+    if _SNAPSHOT_META.exists():
+        try:
+            meta = json.loads(_SNAPSHOT_META.read_text())
+            ts = meta.get("fetched_at")
+            if ts:
+                # strip any trailing 'Z' / timezone offset to a naive UTC datetime
+                ts_clean = ts.replace("Z", "+00:00")
+                fetched_at = datetime.fromisoformat(ts_clean).replace(tzinfo=None)
+        except (ValueError, KeyError) as exc:
+            logger.warning("Could not parse snapshot meta %s: %s", _SNAPSHOT_META, exc)
+    return BessListResult(
+        states=states,
+        source="snapshot",
+        fetched_at=fetched_at,
+    )
+
+
+async def fetch_bess_list() -> BessListResult:
     """
-    Return the BESS list from the AEMO generation information XLSX.
+    Return the BESS list, trying live AEMO → GitHub mirror → bundled snapshot.
 
-    Uses a 24-hour in-memory cache.  Falls back to the static bess_list.json
-    if the download or parse fails and no cached data is available.
+    Uses a 24-hour in-memory cache (per source). The result carries provenance
+    so the API can surface a warning when serving non-live data.
     """
     global _cache
 
@@ -179,32 +262,64 @@ async def fetch_bess_list() -> dict[str, list[dict]]:
 
     # Return cached data if still fresh
     if _cache is not None:
-        parsed, fetched_at = _cache
-        if now - fetched_at < CACHE_TTL:
-            logger.debug("Returning cached BESS list (age %s)", now - fetched_at)
-            return parsed
-
-    # Attempt download
+        cached, cached_at = _cache
+        if now - cached_at < CACHE_TTL:
+            logger.debug("Returning cached BESS list (age %s, source=%s)",
+                         now - cached_at, cached.source)
+            return cached
+
+    async with httpx.AsyncClient(timeout=30, follow_redirects=True) as client:
+        # 1. Live AEMO
+        try:
+            logger.info("Downloading AEMO XLSX (live): %s", XLSX_URL)
+            states = await _download_and_parse(XLSX_URL, client)
+            result = BessListResult(states=states, source="live", fetched_at=now)
+            _cache = (result, now)
+            return result
+        except Exception as exc:
+            logger.warning("Live AEMO XLSX fetch failed: %s", exc)
+
+        # 2. GitHub raw mirror (refreshed daily by CI)
+        try:
+            logger.info("Downloading AEMO XLSX (mirror): %s", XLSX_MIRROR_URL)
+            states = await _download_and_parse(XLSX_MIRROR_URL, client)
+            result = BessListResult(
+                states=states,
+                source="mirror",
+                fetched_at=now,
+                warnings=[
+                    "Live AEMO BESS list is unavailable; showing a copy mirrored "
+                    "from GitHub (updated daily)."
+                ],
+            )
+            _cache = (result, now)
+            return result
+        except Exception as exc:
+            logger.warning("Mirror XLSX fetch failed: %s", exc)
+
+    # 3. Bundled snapshot — last resort. Always serve something rather than 5xx.
     try:
-        logger.info("Downloading AEMO generation information XLSX from %s", XLSX_URL)
-        async with httpx.AsyncClient(timeout=30, follow_redirects=True) as client:
-            resp = await client.get(XLSX_URL)
-            resp.raise_for_status()
-
-        parsed = _parse_xlsx(resp.content)
-        _cache = (parsed, now)
-        return parsed
-
-    except Exception as exc:
-        logger.warning("Failed to fetch/parse AEMO gen info XLSX: %s", exc)
-
-        # Return stale cache rather than nothing
-        if _cache is not None:
-            logger.warning("Returning stale BESS list cache (age %s)", now - _cache[1])
-            return _cache[0]
-
-        # Last resort: return static fallback file
-        logger.warning("Falling back to static bess_list.json")
-        import json
-        fallback = Path(__file__).parent.parent / "data" / "bess_list.json"
-        return json.loads(fallback.read_text())
+        snapshot = _load_snapshot()
+        age = now - snapshot.fetched_at
+        days = max(age.days, 0)
+        snapshot.warnings = [
+            f"Live AEMO BESS list and mirror are both unavailable; "
+            f"showing bundled snapshot from {snapshot.fetched_at:%Y-%m-%d} "
+            f"({days} day{'s' if days != 1 else ''} old)."
+        ]
+        _cache = (snapshot, now)
+        logger.warning("Falling back to bundled snapshot (age %s)", age)
+        return snapshot
+    except FileNotFoundError:
+        logger.error("No bundled snapshot at %s", _SNAPSHOT_JSON)
+        # Return an empty result rather than raising — frontend handles
+        # empty states gracefully and the warning explains why.
+        return BessListResult(
+            states={s: [] for s in REGION_TO_STATE.values()},
+            source="snapshot",
+            fetched_at=now,
+            warnings=[
+                "BESS list is currently unavailable from AEMO and no local "
+                "snapshot is bundled. Try again later."
+            ],
+        )

app/static/css/style.css

@@ -303,8 +303,9 @@ select:disabled, input:disabled {
   gap: 0.6rem;
   align-items: flex-start;
 }
-.alert-error { background: rgba(239,68,68,0.15); border: 1px solid rgba(239,68,68,0.4); color: #fca5a5; }
-.alert-info  { background: rgba(56,189,248,0.1);  border: 1px solid rgba(56,189,248,0.3); color: var(--accent); }
+.alert-error   { background: rgba(239,68,68,0.15);  border: 1px solid rgba(239,68,68,0.4); color: #fca5a5; }
+.alert-info    { background: rgba(56,189,248,0.1);   border: 1px solid rgba(56,189,248,0.3); color: var(--accent); }
+.alert-warning { background: rgba(217,119,6,0.15);   border: 1px solid rgba(217,119,6,0.45); color: #fcd34d; }
 
 .hidden { display: none !important; }
 

app/static/index.html

@@ -21,6 +21,9 @@
 
 <main>
 
+  <!-- ── BESS list staleness banner (shown only when /api/bess returns a non-live source) ── -->
+  <div id="bess-list-banner" class="alert alert-warning hidden" role="status"></div>
+
   <!-- ── Control panel ── -->
   <section class="panel">
     <h2>Select Data</h2>
@@ -54,7 +57,7 @@
               <input type="checkbox" id="chk-scada" checked />
               4-second SCADA
             </label>
-            <button class="about-toggle" data-tip="4-sec power output &amp; quality flags · PUBLIC_NEXT_DAY_FPPMW · from 28 Feb 2025" data-about="scada">▶ About</button>
+            <button class="about-toggle" data-tip="4-sec power output &amp; quality flags · PUBLIC_NEXT_DAY_FPPMW · from 29 Apr 2025" data-about="scada">▶ About</button>
           </div>
 
           <div class="check-row-header">
@@ -62,7 +65,7 @@
               <input type="checkbox" id="chk-energy" checked />
               5-minute Dispatch
             </label>
-            <button class="about-toggle" data-tip="5-min energy storage &amp; dispatch MW · DISPATCH_UNIT_SOLUTION · from 11 Feb 2025" data-about="energy">▶ About</button>
+            <button class="about-toggle" data-tip="5-min energy storage &amp; dispatch MW · DISPATCH_UNIT_SOLUTION · from 1 Apr 2025" data-about="energy">▶ About</button>
           </div>
 
         </div>
@@ -195,7 +198,7 @@
               Extracted from AEMO's <strong>DISPATCH_UNIT_SOLUTION</strong> table and published
               the following day via NEMWEB. Data is fetched live from:
               <a href="https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/" target="_blank" rel="noopener">Current directory</a>
-              (individual daily files, 11 Feb 2025 – yesterday) or the
+              (individual daily files, recent months) or the
               <a href="https://www.nemweb.com.au/REPORTS/ARCHIVE/Next_Day_Dispatch/" target="_blank" rel="noopener">Archive directory</a>
               (monthly bundles for older dates, accessed via HTTP Range requests).
             </p>
@@ -253,13 +256,14 @@
       via NEMWEB:<br/><br/>
       <strong>4-second SCADA data</strong> — from
       <a href="https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/" target="_blank" rel="noopener">NEMWEB FPPDAILY</a>,
-      available from <strong>28 February 2025</strong> when the
+      published as part of the
       <a href="https://aemo.com.au/initiatives/major-programs/frequency-performance-payments-project" target="_blank" rel="noopener">Frequency Performance Payments (FPP)</a>
-      scheme commenced non-financial operation.<br/><br/>
+      scheme (commenced 28 February 2025). Currently fetchable from <strong>29 April 2025</strong>;
+      earlier daily bundles have been rolled off the AEMO archive.<br/><br/>
       <strong>5-minute dispatch energy data</strong> — from
       <a href="https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/" target="_blank" rel="noopener">NEMWEB Next_Day_Dispatch</a>,
-      available from <strong>11 February 2025</strong>. Provides initial energy storage (MWh)
-      and initial MW dispatch for each 5-minute interval.<br/><br/>
+      with the <code>INITIAL_ENERGY_STORAGE</code> column added on 11 February 2025.
+      Currently fetchable from <strong>1 April 2025</strong>; earlier monthly bundles have been rolled off.<br/><br/>
       Data is fetched live from AEMO on each request — no data is stored on this server.
       Maximum request window: <strong>1 day</strong> per query.
     </p>

app/static/js/app.js

@@ -8,8 +8,8 @@ const API = '';   // same origin
 let bessList        = {};
 let qualityFlags    = {};
 let cutoverDate     = '2026-01-11';   // SCADA current/archive cutover (from /api/info)
-let scadaStartDate  = '2025-02-28';   // from /api/info
-let dispatchStartDate = '2025-02-11'; // from /api/info
+let scadaStartDate  = '2025-04-29';   // from /api/info (NEMWEB archive starts here)
+let dispatchStartDate = '2025-04-01'; // from /api/info (NEMWEB archive starts here)
 let appEstimates    = {
   current:          { seconds: 120, sample_count: 0, is_default: true },
   archive:          { seconds: 480, sample_count: 0, is_default: true },
@@ -61,8 +61,9 @@ async function init() {
   } catch (_) {}
 
   // Load BESS list and quality flags in parallel
+  let bessResp;
   try {
-    [bessList, qualityFlags] = await Promise.all([
+    [bessResp, qualityFlags] = await Promise.all([
       fetch(`${API}/api/bess`).then(r => r.json()),
       fetch(`${API}/api/quality-flags`).then(r => r.json()),
     ]);
@@ -70,6 +71,8 @@ async function init() {
     showError('Failed to load BESS list. Please refresh the page.');
     return;
   }
+  bessList = bessResp.states || {};
+  renderBessListBanner(bessResp);
 
   // Populate state selector
   Object.keys(bessList).sort().forEach(state => {
@@ -97,7 +100,7 @@ async function init() {
       html: `
         <p class="about-source">
           Source: <strong>PUBLIC_NEXT_DAY_FPPMW</strong> (FPP Daily)
-          &nbsp;·&nbsp; Available from <strong>28 Feb 2025</strong>
+          &nbsp;·&nbsp; Available from <strong>29 Apr 2025</strong>
         </p>
         <div class="col-defs">
           <div class="col-def">
@@ -123,7 +126,7 @@ async function init() {
       html: `
         <p class="about-source">
           Source: <strong>DISPATCH_UNIT_SOLUTION</strong> via Next_Day_Dispatch
-          &nbsp;·&nbsp; Available from <strong>11 Feb 2025</strong>
+          &nbsp;·&nbsp; Available from <strong>1 Apr 2025</strong>
         </p>
         <div class="col-defs">
           <div class="col-def">
@@ -349,6 +352,31 @@ async function fetchEnergy(duid, date) {
   return resp.json();
 }
 
+/* ── BESS-list staleness banner ── */
+function renderBessListBanner(resp) {
+  const box = document.getElementById('bess-list-banner');
+  if (!box) return;
+  const warnings = (resp && resp.warnings) || [];
+  if (!warnings.length) {
+    box.classList.add('hidden');
+    box.innerHTML = '';
+    return;
+  }
+  // One line per warning, prefixed with a warning icon.
+  box.innerHTML = warnings.map(w =>
+    `<div>⚠️ ${escapeHtml(w)}</div>`
+  ).join('');
+  box.classList.remove('hidden');
+}
+
+function escapeHtml(s) {
+  return String(s)
+    .replace(/&/g, '&amp;')
+    .replace(/</g, '&lt;')
+    .replace(/>/g, '&gt;')
+    .replace(/"/g, '&quot;');
+}
+
 /* ── Warnings render ── */
 function renderWarnings(warnings) {
   let box = document.getElementById('warnings-box');

scripts/refresh_bess_list.py

@@ -0,0 +1,73 @@
+#!/usr/bin/env python3
+"""
+Refresh the bundled BESS list snapshot.
+
+Downloads the AEMO Generation Information XLSX, parses it into the same
+shape served by /api/bess, and writes two files into app/data/:
+
+  bess_list.json      — parsed snapshot used as runtime fallback when the
+                        live AEMO fetch is blocked (e.g. WAF 403 from the
+                        HuggingFace egress IP).
+  aemo_gen_info.xlsx  — the raw XLSX bytes, so gen_info_fetcher.py can also
+                        fall back to fetching it from raw.githubusercontent.com
+                        and parse a fresher copy at runtime.
+
+Run from the repo root:
+    python scripts/refresh_bess_list.py
+
+Intended to be invoked by a scheduled GitHub Action (see
+.github/workflows/refresh_bess_list.yml).
+"""
+from __future__ import annotations
+
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+import httpx
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(REPO_ROOT))
+
+from app.services.gen_info_fetcher import XLSX_URL, _BROWSER_HEADERS, _parse_xlsx
+
+DATA_DIR = REPO_ROOT / "app" / "data"
+JSON_OUT = DATA_DIR / "bess_list.json"
+XLSX_OUT = DATA_DIR / "aemo_gen_info.xlsx"
+META_OUT = DATA_DIR / "bess_list_meta.json"
+
+
+def main() -> int:
+    print(f"Downloading {XLSX_URL}")
+    with httpx.Client(timeout=60, follow_redirects=True) as client:
+        resp = client.get(XLSX_URL, headers=_BROWSER_HEADERS)
+        resp.raise_for_status()
+    xlsx_bytes = resp.content
+    print(f"Got {len(xlsx_bytes):,} bytes")
+
+    parsed = _parse_xlsx(xlsx_bytes)
+    total = sum(len(v) for v in parsed.values())
+    print(f"Parsed {total} BESS units across {sum(1 for v in parsed.values() if v)} states")
+    if total == 0:
+        print("ERROR: parsed XLSX yielded zero BESS units — refusing to overwrite snapshot")
+        return 1
+
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    XLSX_OUT.write_bytes(xlsx_bytes)
+    JSON_OUT.write_text(json.dumps(parsed, indent=2, sort_keys=True) + "\n")
+    META_OUT.write_text(json.dumps({
+        "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
+        "source_url": XLSX_URL,
+        "xlsx_bytes": len(xlsx_bytes),
+        "unit_count": total,
+    }, indent=2) + "\n")
+
+    print(f"Wrote {JSON_OUT.relative_to(REPO_ROOT)}")
+    print(f"Wrote {XLSX_OUT.relative_to(REPO_ROOT)}")
+    print(f"Wrote {META_OUT.relative_to(REPO_ROOT)}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())