Spaces:
Running
Running
File size: 6,883 Bytes
fd0bd3b 721d6bd fd0bd3b 01f0696 fd0bd3b 01f0696 fd0bd3b 01f0696 0d542b0 01f0696 0d542b0 01f0696 318b267 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 | ---
title: NEM Battery SCADA Data Explorer
emoji: 🔋
colorFrom: blue
colorTo: green
sdk: docker
pinned: false
license: mit
short_description: BESS SCADA & dispatch data explorer for the NEM
---
# 🔋 NEM Battery SCADA & Energy Data Explorer
An interactive web app for exploring high-frequency power and energy storage data for Battery Energy Storage Systems (BESS) in Australia's National Electricity Market (NEM). Data is fetched live from AEMO NEMWEB on every request — nothing is stored on the server.
---
## Datasets
### 4-second SCADA — `PUBLIC_NEXT_DAY_FPPMW`
- **Source:** [AEMO NEMWEB FPPDAILY](https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/)
- **Available from:** 29 April 2025 (the FPP scheme itself commenced 28 February 2025, but earlier daily bundles have been rolled off the AEMO NEMWEB archive)
- **Granularity:** one row per 4-second grid slot
| Column | Description |
|--------|-------------|
| `INTERVAL_DATETIME` | Target 4-second grid slot (YYYY/MM/DD HH:MM:SS) |
| `MEASUREMENT_DATETIME` | Actual SCADA recording timestamp (may differ slightly from interval) |
| `MEASURED_MW` | Instantaneous power (MW). Positive = discharging, negative = charging |
| `MW_QUALITY_FLAG` | Data quality: **0** Good · **1** Substituted · **2** Bad · **3** Manual override |
### 5-minute Dispatch — `DISPATCH_UNIT_SOLUTION`
- **Source:** [AEMO NEMWEB Next_Day_Dispatch](https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/)
- **Available from:** 1 April 2025 (the `INITIAL_ENERGY_STORAGE` column was added on 11 February 2025, but earlier monthly bundles have been rolled off the AEMO NEMWEB archive)
- **Granularity:** one row per 5-minute dispatch interval
| Column | Description |
|--------|-------------|
| `SETTLEMENTDATE` | End of the 5-minute interval (AEST) |
| `INITIALMW` | Actual power output (MW) at interval start — the dispatch baseline |
| `INITIAL_ENERGY_STORAGE` | State of Energy (SoE) at interval start (MWh) |
| `ENERGY_STORAGE` | Target SoE at interval end (MWh), accounting for scheduled dispatch |
---
## Features
- **Interactive charts** — zoomable/pannable Plotly.js time-series for both datasets
- **Quality flag visualisation** — colour-coded SCADA points (green / orange / red / grey)
- **Summary statistics** — min, max, mean, std dev, and flag breakdown per query
- **Downloads** — CSV and Parquet for both datasets (Parquet recommended for Python users)
- **Live BESS list** — fetched from AEMO NEM Generation Information (24 h cache), with fallback to bundled JSON
- **26 BESS units** across NSW, VIC, QLD, SA
---
## Tech Stack
| Layer | Technology |
|-------|------------|
| Backend | [FastAPI](https://fastapi.tiangolo.com/) + [Uvicorn](https://www.uvicorn.org/) (Python 3.11) |
| Data processing | [Polars](https://pola.rs/) |
| Remote ZIP access | [remotezip](https://github.com/gtsystem/python-remotezip) (HTTP Range requests for archive bundles) |
| Frontend | Vanilla HTML / CSS / JavaScript |
| Charts | [Plotly.js](https://plotly.com/javascript/) |
| Deployment | Docker → [HuggingFace Spaces](https://huggingface.co/spaces) |
---
## API Endpoints
| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/bess` | BESS units grouped by state |
| `GET` | `/api/data` | 4-second SCADA data (JSON) |
| `GET` | `/api/download/csv` | SCADA data as CSV |
| `GET` | `/api/download/parquet` | SCADA data as Parquet |
| `GET` | `/api/energy-data` | 5-minute dispatch data (JSON) |
| `GET` | `/api/download/energy-csv` | Dispatch data as CSV |
| `GET` | `/api/download/energy-parquet` | Dispatch data as Parquet |
| `GET` | `/api/quality-flags` | `MW_QUALITY_FLAG` code descriptions |
| `GET` | `/api/info` | App metadata (data start dates, timing estimates) |
| `GET` | `/api/analytics` | Request analytics (requires `?token=`) |
**Common query parameters:** `duid=HORNSDALE_PWR1&date=2025-06-01`
---
## Running Locally
```bash
# 1. Clone
git clone https://github.com/pourmousavi/BESS-SCADA-Data.git
cd BESS-SCADA-Data
# 2. Install dependencies
pip install -r requirements.txt
# 3. Run
uvicorn app.main:app --reload --port 8000
```
Then open [http://localhost:8000](http://localhost:8000).
**Optional environment variable:**
| Variable | Default | Description |
|----------|---------|-------------|
| `ANALYTICS_TOKEN` | `changeme` | Token for the `/api/analytics` admin endpoint |
### Docker
```bash
docker build -t bess-explorer .
docker run -p 7860:7860 bess-explorer
```
---
## Project Structure
```
├── app/
│ ├── main.py # FastAPI app & static file serving
│ ├── config.py # AEMO URLs, date constants, timeouts
│ ├── routers/api.py # All REST endpoints
│ ├── services/
│ │ ├── aemo_fetcher.py # Downloads FPPDAILY ZIPs from NEMWEB
│ │ ├── dispatch_fetcher.py # Downloads Next_Day_Dispatch ZIPs
│ │ ├── data_processor.py # Polars: filter & transform SCADA CSV
│ │ ├── dispatch_processor.py# Polars: filter & transform dispatch CSV
│ │ ├── gen_info_fetcher.py # Fetches live BESS list from AEMO XLSX
│ │ └── analytics.py # SQLite request logging
│ ├── data/
│ │ ├── bess_list.json # Fallback static BESS list
│ │ └── quality_flags.json # MW_QUALITY_FLAG descriptions
│ └── static/ # Frontend (HTML + CSS + JS)
├── Dockerfile
└── requirements.txt
```
---
## Data Notes
- **Trading day boundary:** NEM days run 04:00–04:00 AEST, so overnight queries span two calendar dates.
- **FPPDAILY archive format:** Files before 11 Jan 2026 are in monthly/weekly bundle ZIPs (accessed via HTTP Range requests); newer files are individual daily ZIPs.
- **Nested ZIPs:** FPPDAILY archives contain outer ZIP → inner ZIP → CSV. All extraction is done in-memory.
- **FPPMW half-day files:** From March 2025, each NEM day is split into two 12-hour CSV segments published the following day.
- **No data storage:** All data is streamed from AEMO NEMWEB per request. No user data is retained.
---
## Disclaimer
Data is retrieved as-is from [AEMO NEMWEB](https://www.aemo.com.au) and is subject to change, correction, or unavailability without notice. This tool is provided for informational and research purposes only. Not affiliated with AEMO.
**Data source 1:** [AEMO NEMWEB FPPDAILY Current](https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/)
**Data source 2:** [AEMO NEMWEB FPPDAILY Archive](https://www.nemweb.com.au/REPORTS/ARCHIVE/FPPDAILY/)
**Data source 3:** [AEMO NEMWEB Next Day Dispatch Current](https://nemweb.com.au/Reports/Current/Next_Day_Dispatch/)
**Data source 4:** [AEMO NEMWEB Next Day Dispatch Archive](https://www.nemweb.com.au/REPORTS/ARCHIVE/Next_Day_Dispatch/)
|