README.md

metadata

title: NEM Battery SCADA Data Explorer
emoji: 🔋
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
license: mit
short_description: BESS SCADA & dispatch data explorer for the NEM

🔋 NEM Battery SCADA & Energy Data Explorer

An interactive web app for exploring high-frequency power and energy storage data for Battery Energy Storage Systems (BESS) in Australia's National Electricity Market (NEM). Data is fetched live from AEMO NEMWEB on every request — nothing is stored on the server.

---

Datasets

4-second SCADA — PUBLIC_NEXT_DAY_FPPMW

• Source: AEMO NEMWEB FPPDAILY
• Available from: 29 April 2025 (the FPP scheme itself commenced 28 February 2025, but earlier daily bundles have been rolled off the AEMO NEMWEB archive)
• Granularity: one row per 4-second grid slot

Column	Description
INTERVAL_DATETIME	Target 4-second grid slot (YYYY/MM/DD HH:MM:SS)
MEASUREMENT_DATETIME	Actual SCADA recording timestamp (may differ slightly from interval)
MEASURED_MW	Instantaneous power (MW). Positive = discharging, negative = charging
MW_QUALITY_FLAG	Data quality: 0 Good · 1 Substituted · 2 Bad · 3 Manual override

5-minute Dispatch — DISPATCH_UNIT_SOLUTION

• Source: AEMO NEMWEB Next_Day_Dispatch
• Available from: 1 April 2025 (the INITIAL_ENERGY_STORAGE column was added on 11 February 2025, but earlier monthly bundles have been rolled off the AEMO NEMWEB archive)
• Granularity: one row per 5-minute dispatch interval

Column	Description
SETTLEMENTDATE	End of the 5-minute interval (AEST)
INITIALMW	Actual power output (MW) at interval start — the dispatch baseline
INITIAL_ENERGY_STORAGE	State of Energy (SoE) at interval start (MWh)
ENERGY_STORAGE	Target SoE at interval end (MWh), accounting for scheduled dispatch

---

Features

• Interactive charts — zoomable/pannable Plotly.js time-series for both datasets
• Quality flag visualisation — colour-coded SCADA points (green / orange / red / grey)
• Summary statistics — min, max, mean, std dev, and flag breakdown per query
• Downloads — CSV and Parquet for both datasets (Parquet recommended for Python users)
• Live BESS list — fetched from AEMO NEM Generation Information (24 h cache), with fallback to bundled JSON
• 26 BESS units across NSW, VIC, QLD, SA

---

Tech Stack

Layer	Technology
Backend	FastAPI + Uvicorn (Python 3.11)
Data processing	Polars
Remote ZIP access	remotezip (HTTP Range requests for archive bundles)
Frontend	Vanilla HTML / CSS / JavaScript
Charts	Plotly.js
Deployment	Docker → HuggingFace Spaces

---

API Endpoints

Method	Path	Description
GET	/api/bess	BESS units grouped by state
GET	/api/data	4-second SCADA data (JSON)
GET	/api/download/csv	SCADA data as CSV
GET	/api/download/parquet	SCADA data as Parquet
GET	/api/energy-data	5-minute dispatch data (JSON)
GET	/api/download/energy-csv	Dispatch data as CSV
GET	/api/download/energy-parquet	Dispatch data as Parquet
GET	/api/quality-flags	MW_QUALITY_FLAG code descriptions
GET	/api/info	App metadata (data start dates, timing estimates)
GET	/api/analytics	Request analytics (requires ?token=)

Common query parameters: duid=HORNSDALE_PWR1&date=2025-06-01

---

Running Locally

# 1. Clone
git clone https://github.com/pourmousavi/BESS-SCADA-Data.git
cd BESS-SCADA-Data

# 2. Install dependencies
pip install -r requirements.txt

# 3. Run
uvicorn app.main:app --reload --port 8000

Then open http://localhost:8000.

Optional environment variable:

Variable	Default	Description
ANALYTICS_TOKEN	changeme	Token for the /api/analytics admin endpoint

Docker

docker build -t bess-explorer .
docker run -p 7860:7860 bess-explorer

---

Project Structure

├── app/
│   ├── main.py                  # FastAPI app & static file serving
│   ├── config.py                # AEMO URLs, date constants, timeouts
│   ├── routers/api.py           # All REST endpoints
│   ├── services/
│   │   ├── aemo_fetcher.py      # Downloads FPPDAILY ZIPs from NEMWEB
│   │   ├── dispatch_fetcher.py  # Downloads Next_Day_Dispatch ZIPs
│   │   ├── data_processor.py    # Polars: filter & transform SCADA CSV
│   │   ├── dispatch_processor.py# Polars: filter & transform dispatch CSV
│   │   ├── gen_info_fetcher.py  # Fetches live BESS list from AEMO XLSX
│   │   └── analytics.py         # SQLite request logging
│   ├── data/
│   │   ├── bess_list.json       # Fallback static BESS list
│   │   └── quality_flags.json   # MW_QUALITY_FLAG descriptions
│   └── static/                  # Frontend (HTML + CSS + JS)
├── Dockerfile
└── requirements.txt

---

Data Notes

• Trading day boundary: NEM days run 04:00–04:00 AEST, so overnight queries span two calendar dates.
• FPPDAILY archive format: Files before 11 Jan 2026 are in monthly/weekly bundle ZIPs (accessed via HTTP Range requests); newer files are individual daily ZIPs.
• Nested ZIPs: FPPDAILY archives contain outer ZIP → inner ZIP → CSV. All extraction is done in-memory.
• FPPMW half-day files: From March 2025, each NEM day is split into two 12-hour CSV segments published the following day.
• No data storage: All data is streamed from AEMO NEMWEB per request. No user data is retained.

---

Disclaimer

Data is retrieved as-is from AEMO NEMWEB and is subject to change, correction, or unavailability without notice. This tool is provided for informational and research purposes only. Not affiliated with AEMO. Data source 1: AEMO NEMWEB FPPDAILY Current Data source 2: AEMO NEMWEB FPPDAILY Archive Data source 3: AEMO NEMWEB Next Day Dispatch Current Data source 4: AEMO NEMWEB Next Day Dispatch Archive