Spaces:
Running
Running
| title: NEM Battery SCADA Data Explorer | |
| emoji: 🔋 | |
| colorFrom: blue | |
| colorTo: green | |
| sdk: docker | |
| pinned: false | |
| license: mit | |
| short_description: BESS SCADA & dispatch data explorer for the NEM | |
| # 🔋 NEM Battery SCADA & Energy Data Explorer | |
| An interactive web app for exploring high-frequency power and energy storage data for Battery Energy Storage Systems (BESS) in Australia's National Electricity Market (NEM). Data is fetched live from AEMO NEMWEB on every request — nothing is stored on the server. | |
| --- | |
| ## Datasets | |
| ### 4-second SCADA — `PUBLIC_NEXT_DAY_FPPMW` | |
| - **Source:** [AEMO NEMWEB FPPDAILY](https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/) | |
| - **Available from:** 29 April 2025 (the FPP scheme itself commenced 28 February 2025, but earlier daily bundles have been rolled off the AEMO NEMWEB archive) | |
| - **Granularity:** one row per 4-second grid slot | |
| | Column | Description | | |
| |--------|-------------| | |
| | `INTERVAL_DATETIME` | Target 4-second grid slot (YYYY/MM/DD HH:MM:SS) | | |
| | `MEASUREMENT_DATETIME` | Actual SCADA recording timestamp (may differ slightly from interval) | | |
| | `MEASURED_MW` | Instantaneous power (MW). Positive = discharging, negative = charging | | |
| | `MW_QUALITY_FLAG` | Data quality: **0** Good · **1** Substituted · **2** Bad · **3** Manual override | | |
| ### 5-minute Dispatch — `DISPATCH_UNIT_SOLUTION` | |
| - **Source:** [AEMO NEMWEB Next_Day_Dispatch](https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/) | |
| - **Available from:** 1 April 2025 (the `INITIAL_ENERGY_STORAGE` column was added on 11 February 2025, but earlier monthly bundles have been rolled off the AEMO NEMWEB archive) | |
| - **Granularity:** one row per 5-minute dispatch interval | |
| | Column | Description | | |
| |--------|-------------| | |
| | `SETTLEMENTDATE` | End of the 5-minute interval (AEST) | | |
| | `INITIALMW` | Actual power output (MW) at interval start — the dispatch baseline | | |
| | `INITIAL_ENERGY_STORAGE` | State of Energy (SoE) at interval start (MWh) | | |
| | `ENERGY_STORAGE` | Target SoE at interval end (MWh), accounting for scheduled dispatch | | |
| --- | |
| ## Features | |
| - **Interactive charts** — zoomable/pannable Plotly.js time-series for both datasets | |
| - **Quality flag visualisation** — colour-coded SCADA points (green / orange / red / grey) | |
| - **Summary statistics** — min, max, mean, std dev, and flag breakdown per query | |
| - **Downloads** — CSV and Parquet for both datasets (Parquet recommended for Python users) | |
| - **Live BESS list** — fetched from AEMO NEM Generation Information (24 h cache), with fallback to bundled JSON | |
| - **26 BESS units** across NSW, VIC, QLD, SA | |
| --- | |
| ## Tech Stack | |
| | Layer | Technology | | |
| |-------|------------| | |
| | Backend | [FastAPI](https://fastapi.tiangolo.com/) + [Uvicorn](https://www.uvicorn.org/) (Python 3.11) | | |
| | Data processing | [Polars](https://pola.rs/) | | |
| | Remote ZIP access | [remotezip](https://github.com/gtsystem/python-remotezip) (HTTP Range requests for archive bundles) | | |
| | Frontend | Vanilla HTML / CSS / JavaScript | | |
| | Charts | [Plotly.js](https://plotly.com/javascript/) | | |
| | Deployment | Docker → [HuggingFace Spaces](https://huggingface.co/spaces) | | |
| --- | |
| ## API Endpoints | |
| | Method | Path | Description | | |
| |--------|------|-------------| | |
| | `GET` | `/api/bess` | BESS units grouped by state | | |
| | `GET` | `/api/data` | 4-second SCADA data (JSON) | | |
| | `GET` | `/api/download/csv` | SCADA data as CSV | | |
| | `GET` | `/api/download/parquet` | SCADA data as Parquet | | |
| | `GET` | `/api/energy-data` | 5-minute dispatch data (JSON) | | |
| | `GET` | `/api/download/energy-csv` | Dispatch data as CSV | | |
| | `GET` | `/api/download/energy-parquet` | Dispatch data as Parquet | | |
| | `GET` | `/api/quality-flags` | `MW_QUALITY_FLAG` code descriptions | | |
| | `GET` | `/api/info` | App metadata (data start dates, timing estimates) | | |
| | `GET` | `/api/analytics` | Request analytics (requires `?token=`) | | |
| **Common query parameters:** `duid=HORNSDALE_PWR1&date=2025-06-01` | |
| --- | |
| ## Running Locally | |
| ```bash | |
| # 1. Clone | |
| git clone https://github.com/pourmousavi/BESS-SCADA-Data.git | |
| cd BESS-SCADA-Data | |
| # 2. Install dependencies | |
| pip install -r requirements.txt | |
| # 3. Run | |
| uvicorn app.main:app --reload --port 8000 | |
| ``` | |
| Then open [http://localhost:8000](http://localhost:8000). | |
| **Optional environment variable:** | |
| | Variable | Default | Description | | |
| |----------|---------|-------------| | |
| | `ANALYTICS_TOKEN` | `changeme` | Token for the `/api/analytics` admin endpoint | | |
| ### Docker | |
| ```bash | |
| docker build -t bess-explorer . | |
| docker run -p 7860:7860 bess-explorer | |
| ``` | |
| --- | |
| ## Project Structure | |
| ``` | |
| ├── app/ | |
| │ ├── main.py # FastAPI app & static file serving | |
| │ ├── config.py # AEMO URLs, date constants, timeouts | |
| │ ├── routers/api.py # All REST endpoints | |
| │ ├── services/ | |
| │ │ ├── aemo_fetcher.py # Downloads FPPDAILY ZIPs from NEMWEB | |
| │ │ ├── dispatch_fetcher.py # Downloads Next_Day_Dispatch ZIPs | |
| │ │ ├── data_processor.py # Polars: filter & transform SCADA CSV | |
| │ │ ├── dispatch_processor.py# Polars: filter & transform dispatch CSV | |
| │ │ ├── gen_info_fetcher.py # Fetches live BESS list from AEMO XLSX | |
| │ │ └── analytics.py # SQLite request logging | |
| │ ├── data/ | |
| │ │ ├── bess_list.json # Fallback static BESS list | |
| │ │ └── quality_flags.json # MW_QUALITY_FLAG descriptions | |
| │ └── static/ # Frontend (HTML + CSS + JS) | |
| ├── Dockerfile | |
| └── requirements.txt | |
| ``` | |
| --- | |
| ## Data Notes | |
| - **Trading day boundary:** NEM days run 04:00–04:00 AEST, so overnight queries span two calendar dates. | |
| - **FPPDAILY archive format:** Files before 11 Jan 2026 are in monthly/weekly bundle ZIPs (accessed via HTTP Range requests); newer files are individual daily ZIPs. | |
| - **Nested ZIPs:** FPPDAILY archives contain outer ZIP → inner ZIP → CSV. All extraction is done in-memory. | |
| - **FPPMW half-day files:** From March 2025, each NEM day is split into two 12-hour CSV segments published the following day. | |
| - **No data storage:** All data is streamed from AEMO NEMWEB per request. No user data is retained. | |
| --- | |
| ## Disclaimer | |
| Data is retrieved as-is from [AEMO NEMWEB](https://www.aemo.com.au) and is subject to change, correction, or unavailability without notice. This tool is provided for informational and research purposes only. Not affiliated with AEMO. | |
| **Data source 1:** [AEMO NEMWEB FPPDAILY Current](https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/) | |
| **Data source 2:** [AEMO NEMWEB FPPDAILY Archive](https://www.nemweb.com.au/REPORTS/ARCHIVE/FPPDAILY/) | |
| **Data source 3:** [AEMO NEMWEB Next Day Dispatch Current](https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/) | |
| **Data source 4:** [AEMO NEMWEB Next Day Dispatch Archive](https://www.nemweb.com.au/REPORTS/ARCHIVE/Next_Day_Dispatch/) | |