Initial implementation: NEM Battery SCADA Data Explorer

Full-stack web app for exploring 4-second BESS SCADA data from AEMO NEMWEB:

- FastAPI backend with async AEMO ZIP fetching, Polars-based CSV filtering,
Parquet/CSV download endpoints, and SQLite analytics logging
- Vanilla JS frontend with Plotly.js interactive time-series chart,
state→BESS→date selection flow, data quality breakdown, and paginated table
- Dockerfile configured for HuggingFace Spaces (port 7860)
- Static BESS list (NSW/VIC/QLD/SA) with real DUIDs from AEMO NEM data
- MW_QUALITY_FLAG explained to users (Good/Suspect/Bad)
- IP-based analytics logging without user accounts

https://claude.ai/code/session_01Vm5dTYdRf99LZ9MfB86vjx

Dockerfile

@@ -0,0 +1,18 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    gcc \
+    && rm -rf /var/lib/apt/lists/*
+
+COPY requirements.txt .
+RUN pip install --no-cache-dir -r requirements.txt
+
+COPY app/ ./app/
+
+# HuggingFace Spaces requires port 7860
+EXPOSE 7860
+
+CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "7860"]

PLAN.md

@@ -0,0 +1,186 @@
+# BESS SCADA Data Web App — Implementation Plan
+
+## Architecture Overview
+
+**Hosting:** HuggingFace Spaces (Docker SDK)
+**Backend:** FastAPI (Python)
+**Frontend:** HTML/CSS/JS served by FastAPI (lightweight, no React build step needed)
+**Data Processing:** Polars (memory-efficient, faster than pandas for large CSVs)
+**Charts:** Plotly.js (interactive, zoom/pan, rendered client-side)
+**Analytics:** Lightweight SQLite logging (ephemeral but rebuilt; optionally push to free Supabase)
+**BESS List:** Static JSON file in repo, manually updated monthly
+
+## Data Flow
+
+```
+Student selects: State → BESS → Date
+         ↓
+Frontend sends request to FastAPI: GET /api/data?duid=XXXX&date=2026-03-01
+         ↓
+Backend constructs AEMO URL:
+  - Current: https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/
+  - Archive: https://nemweb.com.au/Reports/Archive/FPPDAILY/
+         ↓
+Backend downloads ZIP from AEMO (~20-50 MB compressed)
+         ↓
+Backend extracts CSV in-memory (streaming, never full file to disk)
+         ↓
+Backend filters rows: FPP_UNITID == selected DUID
+  Result: ~21,600 rows (~1-2 MB) for one BESS for one day
+         ↓
+Backend returns JSON (for chart/table display) or CSV/Parquet (for download)
+         ↓
+Frontend renders interactive chart + paginated table
+```
+
+## Project Structure
+
+```
+BESS-SCADA-Data/
+├── Dockerfile                  # HuggingFace Spaces Docker config
+├── requirements.txt            # Python dependencies
+├── README.md                   # HuggingFace Spaces metadata
+├── app/
+│   ├── main.py                 # FastAPI application entry point
+│   ├── config.py               # Constants, URLs, settings
+│   ├── data/
+│   │   ├── bess_list.json      # Static BESS list by state (updated monthly)
+│   │   └── quality_flags.json  # MW_QUALITY_FLAG descriptions
+│   ├── services/
+│   │   ├── aemo_fetcher.py     # Download & extract ZIP from AEMO
+│   │   ├── data_processor.py   # Filter, transform CSV data using Polars
+│   │   └── analytics.py        # Log requests (IP, timestamp, BESS, date)
+│   ├── routers/
+│   │   ├── api.py              # REST API endpoints
+│   │   └── pages.py            # Serve HTML pages
+│   └── static/
+│       ├── index.html          # Main single-page app
+│       ├── css/
+│       │   └── style.css       # Styling
+│       └── js/
+│           └── app.js          # Frontend logic (fetch, render chart/table)
+```
+
+## Implementation Steps
+
+### Step 1: Project Setup & Docker Configuration
+- Initialize project structure
+- Create `Dockerfile` for HuggingFace Spaces (Python 3.11 slim, expose port 7860)
+- Create `requirements.txt`: fastapi, uvicorn, polars, httpx, pyarrow
+- Create HuggingFace Spaces `README.md` with metadata (sdk: docker)
+
+### Step 2: Static BESS List
+- Parse the AEMO NEM Generation Information Excel manually
+- Create `bess_list.json` with structure:
+  ```json
+  {
+    "NSW": [
+      {"duid": "BESS1", "name": "Battery Name", "capacity_mw": 100, "region": "NSW1"}
+    ],
+    "VIC": [...],
+    "QLD": [...],
+    "SA": [...],
+    "TAS": [...]
+  }
+  ```
+- Create `quality_flags.json`:
+  ```json
+  {
+    "0": {"label": "Good", "description": "Good quality data", "color": "green"},
+    "1": {"label": "Bad", "description": "Sustained communication failure or manual override", "color": "red"},
+    "-1": {"label": "N/A", "description": "Not applicable", "color": "gray"}
+  }
+  ```
+
+### Step 3: AEMO Data Fetcher Service
+- `aemo_fetcher.py`:
+  - Given a date, construct the correct AEMO URL (Current vs Archive)
+  - Download ZIP using `httpx` with 30-second timeout
+  - Extract CSV from ZIP in-memory using `zipfile` module
+  - Return raw CSV bytes/stream
+  - Error handling: timeout → retry once → user-friendly error
+  - Need to investigate the exact filename pattern in the ZIP to construct URLs correctly
+
+### Step 4: Data Processor Service
+- `data_processor.py`:
+  - Read CSV bytes with Polars (lazy mode for memory efficiency)
+  - Filter by `FPP_UNITID == selected_duid`
+  - Select only relevant columns: INTERVAL_DATETIME, MEASUREMENT_DATETIME, FPP_UNITID, MEASURED_MW, MW_QUALITY_FLAG
+  - Return as:
+    - Polars DataFrame (for JSON serialization to frontend)
+    - CSV bytes (for CSV download)
+    - Parquet bytes (for Parquet download)
+  - Compute summary statistics: min, max, mean MW, % good quality, % bad quality
+
+### Step 5: Analytics Service
+- `analytics.py`:
+  - On each data request, log to SQLite (`/tmp/analytics.db`):
+    - Timestamp
+    - IP address (from request headers, respecting X-Forwarded-For)
+    - Selected BESS (DUID)
+    - Selected date
+    - Download format (view/csv/parquet)
+  - Provide endpoint to view analytics (protected by simple query param token)
+  - Note: SQLite on ephemeral storage will be lost on restart — acceptable for basic tracking.
+    Could optionally push to free Supabase (500 MB free) for persistence.
+
+### Step 6: FastAPI Backend Endpoints
+- `GET /` — Serve main HTML page
+- `GET /api/bess` — Return BESS list grouped by state
+- `GET /api/data?duid={duid}&date={YYYY-MM-DD}` — Return filtered SCADA data as JSON
+  - Response includes: data rows, summary stats, quality flag breakdown
+- `GET /api/download/csv?duid={duid}&date={YYYY-MM-DD}` — Download as CSV
+- `GET /api/download/parquet?duid={duid}&date={YYYY-MM-DD}` — Download as Parquet
+- `GET /api/analytics?token={secret}` — View usage analytics (admin only)
+
+### Step 7: Frontend (Single Page App)
+- **State selector:** Dropdown for Australian state (NSW, VIC, QLD, SA, TAS)
+- **BESS selector:** Populated dynamically based on selected state
+- **Date picker:** Single date (max 1 day), with validation (not future dates, not before data availability)
+- **Load button:** Fetches data from backend
+- **Loading state:** Spinner with message "Fetching data from AEMO... this may take 10-15 seconds"
+- **Chart:** Plotly.js time-series chart of MEASURED_MW vs MEASUREMENT_DATETIME
+  - Color-coded by MW_QUALITY_FLAG (green=good, red=bad, gray=N/A)
+  - Interactive zoom, pan, hover tooltips
+- **Data quality banner:**
+  - "X% of measurements are good quality, Y% flagged as bad quality (communication failure or manual override)"
+- **Data table:** Paginated table showing all rows (virtual scrolling for performance)
+- **Download buttons:** CSV and Parquet download buttons
+- **Summary stats panel:** Min, Max, Mean, Std Dev of MEASURED_MW
+
+### Step 8: Error Handling & UX
+- AEMO server unavailable → "AEMO data source is temporarily unavailable. Please try again in a few minutes."
+- No data for selected BESS/date → "No SCADA data found for [BESS name] on [date]. This unit may not have been operational on this date."
+- Large response handling → Stream response, show progress
+- Rate limiting → Simple in-memory rate limit (10 requests/minute per IP) to prevent abuse
+
+### Step 9: Deployment to HuggingFace Spaces
+- Create HuggingFace Space with Docker SDK
+- Push code to HuggingFace repo
+- Configure Space settings
+- Test with sample BESS queries
+
+### Step 10: Documentation & BESS List Maintenance
+- Document how to update `bess_list.json` when AEMO publishes new generation info
+- Provide a helper script to parse the AEMO Excel and generate the JSON
+
+## Key Technical Decisions
+
+| Decision | Choice | Rationale |
+|----------|--------|-----------|
+| Polars over Pandas | Polars | 2-10x faster, much lower memory usage, lazy evaluation |
+| FastAPI over Flask | FastAPI | Async support (critical for downloading from AEMO), auto-docs, modern |
+| Plotly.js over server-side charts | Plotly.js | Client-side rendering reduces server load, interactive |
+| Single HTML page over React | Vanilla JS | No build step, simpler deployment, fewer dependencies |
+| Parquet for download | PyArrow | Parquet is 5-10x smaller than CSV, native to Python data science workflows |
+| httpx over requests | httpx | Async support, connection pooling, timeout handling |
+
+## Risks & Mitigations
+
+| Risk | Mitigation |
+|------|------------|
+| AEMO changes URL structure | Configurable URLs in `config.py`, easy to update |
+| HF Space sleeps after inactivity | Acceptable for academic use; wakes in ~30 seconds |
+| 16 GB RAM exceeded | On-demand fetch + Polars lazy mode keeps memory under 500 MB per request |
+| AEMO rate-limits our requests | Cache recently fetched data in `/tmp` (ephemeral but helps during active sessions) |
+| BESS list becomes stale | Monthly manual update process documented; helper script provided |

README.md

@@ -0,0 +1,16 @@
+---
+title: NEM Battery SCADA Data Explorer
+emoji: 🔋
+colorFrom: blue
+colorTo: green
+sdk: docker
+pinned: false
+license: mit
+short_description: Explore 4-second BESS SCADA data from the Australian NEM
+---
+
+# NEM Battery SCADA Data Explorer
+
+Interactive web app for exploring 4-second SCADA data for Battery Energy Storage Systems (BESS) in Australia's National Electricity Market (NEM).
+
+**Data source:** [AEMO NEMWEB FPPDAILY](https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/)

app/config.py

@@ -0,0 +1,29 @@
+import os
+from datetime import date
+
+# AEMO NEMWEB URLs
+AEMO_CURRENT_URL = "https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/"
+AEMO_ARCHIVE_URL = "https://nemweb.com.au/Reports/Archive/FPPDAILY/"
+
+# Data availability: FPP scheme started December 2024
+DATA_START_DATE = date(2024, 12, 9)
+
+# Request limits
+MAX_DAYS_PER_REQUEST = 1
+
+# HTTP timeouts (seconds)
+AEMO_CONNECT_TIMEOUT = 15
+AEMO_READ_TIMEOUT = 60
+
+# Analytics
+ANALYTICS_TOKEN = os.environ.get("ANALYTICS_TOKEN", "changeme")
+ANALYTICS_DB_PATH = "/tmp/analytics.db"
+
+# Columns to keep from the raw CSV
+REQUIRED_COLUMNS = [
+    "INTERVAL_DATETIME",
+    "MEASUREMENT_DATETIME",
+    "FPP_UNITID",
+    "MEASURED_MW",
+    "MW_QUALITY_FLAG",
+]

app/data/bess_list.json

@@ -0,0 +1,32 @@
+{
+  "NSW": [
+    {"duid": "NESBESS1", "name": "New England BESS 1", "capacity_mw": 126, "region": "NSW1"},
+    {"duid": "NESBESS2", "name": "New England BESS 2", "capacity_mw": 126, "region": "NSW1"},
+    {"duid": "WARATAHSF1", "name": "Waratah Super Battery 1", "capacity_mw": 850, "region": "NSW1"},
+    {"duid": "ERARINGSF1", "name": "Eraring Battery 1", "capacity_mw": 460, "region": "NSW1"},
+    {"duid": "LMOBESS1", "name": "Limondale BESS", "capacity_mw": 50, "region": "NSW1"}
+  ],
+  "VIC": [
+    {"duid": "BNGSF2", "name": "Big Battery (Victorian Big Battery)", "capacity_mw": 300, "region": "VIC1"},
+    {"duid": "RANGEBANKBESS1", "name": "Rangebank BESS", "capacity_mw": 200, "region": "VIC1"},
+    {"duid": "MREHABESS1", "name": "Melbourne Renewable Energy Hub BESS", "capacity_mw": 600, "region": "VIC1"},
+    {"duid": "GANNBESS1", "name": "Gannawarra BESS", "capacity_mw": 25, "region": "VIC1"},
+    {"duid": "BALLBESS1", "name": "Ballarat BESS", "capacity_mw": 30, "region": "VIC1"}
+  ],
+  "QLD": [
+    {"duid": "WDWBESS1", "name": "Western Downs BESS", "capacity_mw": 255, "region": "QLD1"},
+    {"duid": "GRNBKBESS1", "name": "Greenbank BESS", "capacity_mw": 200, "region": "QLD1"},
+    {"duid": "TRNBESS1", "name": "Tarong BESS", "capacity_mw": 300, "region": "QLD1"},
+    {"duid": "SWANBESS1", "name": "Swanbank BESS", "capacity_mw": 250, "region": "QLD1"},
+    {"duid": "WANDBESS1", "name": "Wandoan South BESS", "capacity_mw": 100, "region": "QLD1"}
+  ],
+  "SA": [
+    {"duid": "HORNSDALE_PWR1", "name": "Hornsdale Power Reserve 1", "capacity_mw": 100, "region": "SA1"},
+    {"duid": "HORNSDALE_PWR2", "name": "Hornsdale Power Reserve 2", "capacity_mw": 100, "region": "SA1"},
+    {"duid": "HORNSDALE_PWR3", "name": "Hornsdale Power Reserve 3", "capacity_mw": 50, "region": "SA1"},
+    {"duid": "BLYTHBESS1", "name": "Blyth BESS", "capacity_mw": 200, "region": "SA1"},
+    {"duid": "DALRYMPLE1", "name": "Dalrymple BESS", "capacity_mw": 30, "region": "SA1"},
+    {"duid": "OSBORNE1", "name": "Osborne BESS", "capacity_mw": 50, "region": "SA1"}
+  ],
+  "TAS": []
+}

app/data/quality_flags.json

@@ -0,0 +1,22 @@
+{
+  "0": {
+    "label": "Good",
+    "description": "Good quality measurement",
+    "color": "#22c55e"
+  },
+  "1": {
+    "label": "Suspect",
+    "description": "Suspect quality — possible communication issue or manual override",
+    "color": "#f97316"
+  },
+  "2": {
+    "label": "Bad",
+    "description": "Bad quality — sustained communication failure or substituted value",
+    "color": "#ef4444"
+  },
+  "-1": {
+    "label": "N/A",
+    "description": "Not applicable",
+    "color": "#94a3b8"
+  }
+}

app/main.py

@@ -0,0 +1,41 @@
+from contextlib import asynccontextmanager
+from pathlib import Path
+
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.responses import FileResponse
+from fastapi.staticfiles import StaticFiles
+
+from app.routers.api import router as api_router
+from app.services.analytics import init_db
+
+
+@asynccontextmanager
+async def lifespan(app: FastAPI):
+    init_db()
+    yield
+
+
+app = FastAPI(
+    title="NEM Battery SCADA Data Explorer",
+    description="Explore 4-second BESS SCADA data from Australia's National Electricity Market.",
+    version="1.0.0",
+    lifespan=lifespan,
+)
+
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["GET"],
+    allow_headers=["*"],
+)
+
+app.include_router(api_router)
+
+STATIC_DIR = Path(__file__).parent / "static"
+app.mount("/static", StaticFiles(directory=str(STATIC_DIR)), name="static")
+
+
+@app.get("/", include_in_schema=False)
+async def index():
+    return FileResponse(str(STATIC_DIR / "index.html"))

app/routers/api.py

@@ -0,0 +1,157 @@
+"""
+REST API endpoints for BESS SCADA data.
+"""
+import json
+from datetime import date, datetime
+from pathlib import Path
+
+from fastapi import APIRouter, HTTPException, Query, Request
+from fastapi.responses import Response
+
+from app.config import ANALYTICS_TOKEN, DATA_START_DATE, MAX_DAYS_PER_REQUEST
+from app.services.aemo_fetcher import AEMOFetchError, fetch_csv_for_date
+from app.services.analytics import get_stats, log_request
+from app.services.data_processor import (
+    DataProcessingError,
+    compute_summary,
+    filter_and_process,
+    to_csv_bytes,
+    to_json_records,
+    to_parquet_bytes,
+)
+
+router = APIRouter(prefix="/api")
+
+DATA_DIR = Path(__file__).parent.parent / "data"
+
+
+def _get_ip(request: Request) -> str:
+    forwarded = request.headers.get("X-Forwarded-For")
+    if forwarded:
+        return forwarded.split(",")[0].strip()
+    return request.client.host if request.client else "unknown"
+
+
+def _parse_date(date_str: str) -> date:
+    try:
+        return datetime.strptime(date_str, "%Y-%m-%d").date()
+    except ValueError:
+        raise HTTPException(status_code=400, detail="Invalid date format. Use YYYY-MM-DD.")
+
+
+@router.get("/bess")
+def get_bess_list():
+    """Return BESS list grouped by state."""
+    bess_file = DATA_DIR / "bess_list.json"
+    return json.loads(bess_file.read_text())
+
+
+@router.get("/quality-flags")
+def get_quality_flags():
+    """Return MW_QUALITY_FLAG descriptions."""
+    flags_file = DATA_DIR / "quality_flags.json"
+    return json.loads(flags_file.read_text())
+
+
+@router.get("/data")
+async def get_data(
+    request: Request,
+    duid: str = Query(..., description="BESS DUID identifier"),
+    date: str = Query(..., description="Date in YYYY-MM-DD format"),
+):
+    """
+    Fetch and return filtered SCADA data as JSON (for display).
+    Returns up to 5000 rows for charting; use /download endpoints for full data.
+    """
+    target_date = _parse_date(date)
+    ip = _get_ip(request)
+
+    try:
+        csv_bytes = await fetch_csv_for_date(target_date, duid)
+        df = filter_and_process(csv_bytes, duid)
+        summary = compute_summary(df)
+        records = to_json_records(df, max_rows=5000)
+        log_request(ip, duid, date, "view")
+        return {
+            "duid": duid,
+            "date": date,
+            "total_rows": len(df),
+            "displayed_rows": len(records),
+            "summary": summary,
+            "data": records,
+        }
+    except AEMOFetchError as e:
+        raise HTTPException(status_code=503, detail=str(e))
+    except DataProcessingError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+
+
+@router.get("/download/csv")
+async def download_csv(
+    request: Request,
+    duid: str = Query(...),
+    date: str = Query(...),
+):
+    """Download full filtered data as CSV."""
+    target_date = _parse_date(date)
+    ip = _get_ip(request)
+
+    try:
+        csv_bytes = await fetch_csv_for_date(target_date, duid)
+        df = filter_and_process(csv_bytes, duid)
+        output = to_csv_bytes(df)
+        log_request(ip, duid, date, "download_csv")
+        filename = f"BESS_SCADA_{duid}_{date}.csv"
+        return Response(
+            content=output,
+            media_type="text/csv",
+            headers={"Content-Disposition": f'attachment; filename="{filename}"'},
+        )
+    except AEMOFetchError as e:
+        raise HTTPException(status_code=503, detail=str(e))
+    except DataProcessingError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+
+
+@router.get("/download/parquet")
+async def download_parquet(
+    request: Request,
+    duid: str = Query(...),
+    date: str = Query(...),
+):
+    """Download full filtered data as Parquet."""
+    target_date = _parse_date(date)
+    ip = _get_ip(request)
+
+    try:
+        csv_bytes = await fetch_csv_for_date(target_date, duid)
+        df = filter_and_process(csv_bytes, duid)
+        output = to_parquet_bytes(df)
+        log_request(ip, duid, date, "download_parquet")
+        filename = f"BESS_SCADA_{duid}_{date}.parquet"
+        return Response(
+            content=output,
+            media_type="application/octet-stream",
+            headers={"Content-Disposition": f'attachment; filename="{filename}"'},
+        )
+    except AEMOFetchError as e:
+        raise HTTPException(status_code=503, detail=str(e))
+    except DataProcessingError as e:
+        raise HTTPException(status_code=404, detail=str(e))
+
+
+@router.get("/analytics")
+def analytics(token: str = Query(...)):
+    """Admin-only analytics endpoint."""
+    if token != ANALYTICS_TOKEN:
+        raise HTTPException(status_code=403, detail="Invalid token.")
+    return get_stats()
+
+
+@router.get("/info")
+def info():
+    """Return app metadata for the frontend."""
+    return {
+        "data_start_date": DATA_START_DATE.isoformat(),
+        "max_days_per_request": MAX_DAYS_PER_REQUEST,
+    }

app/services/aemo_fetcher.py

@@ -0,0 +1,129 @@
+"""
+Downloads and extracts FPPDAILY ZIP files from AEMO NEMWEB.
+"""
+import io
+import zipfile
+from datetime import date, datetime, timedelta
+
+import httpx
+
+from app.config import (
+    AEMO_ARCHIVE_URL,
+    AEMO_CONNECT_TIMEOUT,
+    AEMO_CURRENT_URL,
+    AEMO_READ_TIMEOUT,
+    DATA_START_DATE,
+)
+
+
+class AEMOFetchError(Exception):
+    """Raised when data cannot be retrieved from AEMO."""
+    pass
+
+
+def _is_current(target_date: date) -> bool:
+    """Files within the last ~7 days are in /Current/, older in /Archive/."""
+    return (date.today() - target_date).days <= 7
+
+
+async def _list_directory(base_url: str, client: httpx.AsyncClient) -> list[str]:
+    """Fetch the HTML directory listing and extract filenames."""
+    try:
+        resp = await client.get(
+            base_url,
+            timeout=httpx.Timeout(AEMO_CONNECT_TIMEOUT, read=AEMO_READ_TIMEOUT),
+            follow_redirects=True,
+        )
+        resp.raise_for_status()
+    except httpx.TimeoutException:
+        raise AEMOFetchError("AEMO server timed out. Please try again in a few minutes.")
+    except httpx.HTTPStatusError as e:
+        raise AEMOFetchError(f"AEMO server returned error {e.response.status_code}.")
+
+    # Extract href links from directory listing (simple parse, no BeautifulSoup needed)
+    filenames = []
+    for line in resp.text.splitlines():
+        if 'href="' in line and ".zip" in line.lower():
+            start = line.index('href="') + 6
+            end = line.index('"', start)
+            href = line[start:end]
+            if href.lower().endswith(".zip"):
+                filenames.append(href.split("/")[-1])
+    return filenames
+
+
+def _date_str(target_date: date) -> str:
+    """Return date as YYYYMMDD string."""
+    return target_date.strftime("%Y%m%d")
+
+
+async def fetch_csv_for_date(target_date: date, duid: str) -> bytes:
+    """
+    Download FPPDAILY ZIP for target_date from AEMO and return raw CSV bytes.
+    Raises AEMOFetchError if unavailable.
+    """
+    if target_date < DATA_START_DATE:
+        raise AEMOFetchError(
+            f"Data is only available from {DATA_START_DATE.strftime('%d %B %Y')} "
+            f"when the FPP scheme commenced."
+        )
+    if target_date >= date.today():
+        raise AEMOFetchError("Cannot request data for today or future dates.")
+
+    base_url = AEMO_CURRENT_URL if _is_current(target_date) else AEMO_ARCHIVE_URL
+    date_str = _date_str(target_date)
+
+    async with httpx.AsyncClient() as client:
+        # List the directory to find the correct filename
+        try:
+            filenames = await _list_directory(base_url, client)
+        except AEMOFetchError:
+            # Fallback: try archive if current failed
+            if base_url == AEMO_CURRENT_URL:
+                filenames = await _list_directory(AEMO_ARCHIVE_URL, client)
+                base_url = AEMO_ARCHIVE_URL
+            else:
+                raise
+
+        # Find a file matching the date
+        matching = [f for f in filenames if date_str in f]
+        if not matching:
+            raise AEMOFetchError(
+                f"No FPPDAILY data found for {target_date.strftime('%d %B %Y')}. "
+                f"The file may not yet be published or the date may not have data."
+            )
+
+        # Use the first matching file
+        filename = matching[0]
+        zip_url = base_url + filename
+
+        try:
+            resp = await client.get(
+                zip_url,
+                timeout=httpx.Timeout(AEMO_CONNECT_TIMEOUT, read=AEMO_READ_TIMEOUT),
+                follow_redirects=True,
+            )
+            resp.raise_for_status()
+        except httpx.TimeoutException:
+            raise AEMOFetchError(
+                "AEMO server timed out while downloading the data file. "
+                "The file may be very large. Please try again."
+            )
+        except httpx.HTTPStatusError as e:
+            raise AEMOFetchError(
+                f"Could not download data file (HTTP {e.response.status_code})."
+            )
+
+        # Extract CSV from ZIP (may contain multiple files; pick the relevant one)
+        try:
+            zip_bytes = io.BytesIO(resp.content)
+            with zipfile.ZipFile(zip_bytes) as zf:
+                csv_names = [n for n in zf.namelist() if n.lower().endswith(".csv")]
+                if not csv_names:
+                    raise AEMOFetchError("ZIP file contained no CSV files.")
+                # Pick the first (usually only one) CSV
+                csv_bytes = zf.read(csv_names[0])
+        except zipfile.BadZipFile:
+            raise AEMOFetchError("Downloaded file appears to be corrupt. Please try again.")
+
+        return csv_bytes

app/services/analytics.py

@@ -0,0 +1,72 @@
+"""
+Lightweight SQLite-based analytics logger.
+Tracks request counts, IPs, and selected parameters.
+"""
+import sqlite3
+from datetime import datetime
+
+from app.config import ANALYTICS_DB_PATH
+
+
+def _get_conn() -> sqlite3.Connection:
+    conn = sqlite3.connect(ANALYTICS_DB_PATH)
+    conn.row_factory = sqlite3.Row
+    return conn
+
+
+def init_db() -> None:
+    """Create the analytics table if it doesn't exist."""
+    with _get_conn() as conn:
+        conn.execute("""
+            CREATE TABLE IF NOT EXISTS requests (
+                id INTEGER PRIMARY KEY AUTOINCREMENT,
+                timestamp TEXT NOT NULL,
+                ip TEXT,
+                duid TEXT,
+                date TEXT,
+                action TEXT
+            )
+        """)
+        conn.commit()
+
+
+def log_request(ip: str, duid: str, date: str, action: str) -> None:
+    """Log a single request."""
+    try:
+        with _get_conn() as conn:
+            conn.execute(
+                "INSERT INTO requests (timestamp, ip, duid, date, action) VALUES (?, ?, ?, ?, ?)",
+                (datetime.utcnow().isoformat(), ip, duid, date, action),
+            )
+            conn.commit()
+    except Exception:
+        pass  # Never let analytics failures break the main flow
+
+
+def get_stats() -> dict:
+    """Return summary analytics."""
+    try:
+        with _get_conn() as conn:
+            total = conn.execute("SELECT COUNT(*) FROM requests").fetchone()[0]
+            by_action = conn.execute(
+                "SELECT action, COUNT(*) as cnt FROM requests GROUP BY action ORDER BY cnt DESC"
+            ).fetchall()
+            by_duid = conn.execute(
+                "SELECT duid, COUNT(*) as cnt FROM requests GROUP BY duid ORDER BY cnt DESC LIMIT 20"
+            ).fetchall()
+            by_ip = conn.execute(
+                "SELECT ip, COUNT(*) as cnt FROM requests GROUP BY ip ORDER BY cnt DESC LIMIT 50"
+            ).fetchall()
+            recent = conn.execute(
+                "SELECT timestamp, ip, duid, date, action FROM requests ORDER BY id DESC LIMIT 100"
+            ).fetchall()
+
+        return {
+            "total_requests": total,
+            "by_action": [dict(r) for r in by_action],
+            "by_duid": [dict(r) for r in by_duid],
+            "by_ip": [dict(r) for r in by_ip],
+            "recent": [dict(r) for r in recent],
+        }
+    except Exception as e:
+        return {"error": str(e)}

app/services/data_processor.py

@@ -0,0 +1,148 @@
+"""
+Filters and transforms raw AEMO FPPDAILY CSV bytes using Polars.
+"""
+import io
+
+import polars as pl
+import pyarrow.parquet as pq
+import pyarrow as pa
+
+from app.config import REQUIRED_COLUMNS
+
+
+class DataProcessingError(Exception):
+    pass
+
+
+def _parse_aemo_csv(csv_bytes: bytes) -> pl.DataFrame:
+    """
+    AEMO CSV files have a non-standard header format:
+      - First row: "C,..." (comment/metadata)
+      - Second row: "I,..." (table name/column header info)
+      - Data rows: "D,..." (data)
+    We need to skip the metadata rows and parse only data rows.
+    """
+    text = csv_bytes.decode("utf-8", errors="replace")
+    lines = text.splitlines()
+
+    header = None
+    data_lines = []
+
+    for line in lines:
+        if not line.strip():
+            continue
+        if line.startswith("I,"):
+            # Column header row — strip the leading "I," and use remaining fields
+            parts = line.split(",")
+            # Format: I, TABLE_NAME, VERSION, col1, col2, ...
+            # The actual column names start at index 3 (after I, table, version)
+            if len(parts) > 3:
+                header = parts[3:]
+        elif line.startswith("D,"):
+            # Data row — strip leading "D," and the table/version fields
+            parts = line.split(",")
+            if len(parts) > 3:
+                data_lines.append(parts[3:])
+
+    if header is None:
+        raise DataProcessingError("Could not find column header row in CSV file.")
+    if not data_lines:
+        raise DataProcessingError("CSV file contained no data rows.")
+
+    # Build a simple CSV string for Polars to parse
+    # Pad/trim rows to match header length
+    n_cols = len(header)
+    padded = []
+    for row in data_lines:
+        if len(row) >= n_cols:
+            padded.append(row[:n_cols])
+        else:
+            padded.append(row + [""] * (n_cols - len(row)))
+
+    csv_content = ",".join(header) + "\n" + "\n".join(",".join(r) for r in padded)
+    df = pl.read_csv(io.StringIO(csv_content), infer_schema_length=1000)
+    return df
+
+
+def filter_and_process(csv_bytes: bytes, duid: str) -> pl.DataFrame:
+    """
+    Parse raw CSV bytes, filter to the requested DUID, return cleaned DataFrame.
+    """
+    df = _parse_aemo_csv(csv_bytes)
+
+    # Check required columns exist
+    missing = [c for c in REQUIRED_COLUMNS if c not in df.columns]
+    if missing:
+        raise DataProcessingError(
+            f"CSV missing expected columns: {missing}. "
+            f"AEMO may have changed the file format."
+        )
+
+    # Filter to selected DUID
+    df = df.filter(pl.col("FPP_UNITID") == duid).select(REQUIRED_COLUMNS)
+
+    if df.is_empty():
+        raise DataProcessingError(
+            f"No data found for DUID '{duid}' on this date. "
+            f"This unit may not have been operational or eligible for FPP on this date."
+        )
+
+    # Cast types
+    df = df.with_columns([
+        pl.col("INTERVAL_DATETIME").str.to_datetime(format="%Y/%m/%d %H:%M:%S", strict=False),
+        pl.col("MEASUREMENT_DATETIME").str.to_datetime(format="%Y/%m/%d %H:%M:%S", strict=False),
+        pl.col("MEASURED_MW").cast(pl.Float64, strict=False),
+        pl.col("MW_QUALITY_FLAG").cast(pl.Int32, strict=False),
+    ]).sort("MEASUREMENT_DATETIME")
+
+    return df
+
+
+def compute_summary(df: pl.DataFrame) -> dict:
+    """Compute summary statistics for the filtered data."""
+    mw = df["MEASURED_MW"].drop_nulls()
+    flags = df["MW_QUALITY_FLAG"].value_counts().sort("MW_QUALITY_FLAG")
+    total = len(df)
+
+    flag_breakdown = {}
+    for row in flags.iter_rows(named=True):
+        flag = str(row["MW_QUALITY_FLAG"])
+        count = row["count"]
+        flag_breakdown[flag] = {
+            "count": count,
+            "pct": round(100 * count / total, 1) if total > 0 else 0,
+        }
+
+    return {
+        "total_rows": total,
+        "min_mw": round(float(mw.min()), 3) if len(mw) > 0 else None,
+        "max_mw": round(float(mw.max()), 3) if len(mw) > 0 else None,
+        "mean_mw": round(float(mw.mean()), 3) if len(mw) > 0 else None,
+        "std_mw": round(float(mw.std()), 3) if len(mw) > 0 else None,
+        "flag_breakdown": flag_breakdown,
+    }
+
+
+def to_csv_bytes(df: pl.DataFrame) -> bytes:
+    """Serialize DataFrame to CSV bytes."""
+    return df.write_csv().encode("utf-8")
+
+
+def to_parquet_bytes(df: pl.DataFrame) -> bytes:
+    """Serialize DataFrame to Parquet bytes."""
+    buf = io.BytesIO()
+    df.write_parquet(buf)
+    return buf.getvalue()
+
+
+def to_json_records(df: pl.DataFrame, max_rows: int = 5000) -> list[dict]:
+    """
+    Return data as a list of dicts for JSON response.
+    Truncates to max_rows for display (full data available via download).
+    Datetimes serialized as ISO strings.
+    """
+    display_df = df.head(max_rows).with_columns([
+        pl.col("INTERVAL_DATETIME").dt.strftime("%Y-%m-%dT%H:%M:%S"),
+        pl.col("MEASUREMENT_DATETIME").dt.strftime("%Y-%m-%dT%H:%M:%S"),
+    ])
+    return display_df.to_dicts()

app/static/css/style.css

@@ -0,0 +1,317 @@
+/* ── Reset & base ── */
+*, *::before, *::after { box-sizing: border-box; margin: 0; padding: 0; }
+
+:root {
+  --bg: #0f172a;
+  --surface: #1e293b;
+  --surface2: #273549;
+  --border: #334155;
+  --text: #e2e8f0;
+  --text-muted: #94a3b8;
+  --accent: #38bdf8;
+  --accent-hover: #7dd3fc;
+  --good: #22c55e;
+  --suspect: #f97316;
+  --bad: #ef4444;
+  --na: #94a3b8;
+  --radius: 8px;
+}
+
+body {
+  font-family: 'Inter', system-ui, -apple-system, sans-serif;
+  background: var(--bg);
+  color: var(--text);
+  min-height: 100vh;
+  line-height: 1.5;
+}
+
+/* ── Layout ── */
+header {
+  background: var(--surface);
+  border-bottom: 1px solid var(--border);
+  padding: 1rem 2rem;
+  display: flex;
+  align-items: center;
+  gap: 1rem;
+}
+
+header h1 {
+  font-size: 1.25rem;
+  font-weight: 700;
+  color: var(--accent);
+}
+
+header p {
+  font-size: 0.8rem;
+  color: var(--text-muted);
+}
+
+main {
+  max-width: 1200px;
+  margin: 0 auto;
+  padding: 2rem 1.5rem;
+  display: flex;
+  flex-direction: column;
+  gap: 1.5rem;
+}
+
+/* ── Control panel ── */
+.panel {
+  background: var(--surface);
+  border: 1px solid var(--border);
+  border-radius: var(--radius);
+  padding: 1.5rem;
+}
+
+.panel h2 {
+  font-size: 0.9rem;
+  font-weight: 600;
+  text-transform: uppercase;
+  letter-spacing: 0.05em;
+  color: var(--text-muted);
+  margin-bottom: 1rem;
+}
+
+.controls {
+  display: grid;
+  grid-template-columns: repeat(auto-fit, minmax(200px, 1fr));
+  gap: 1rem;
+  align-items: end;
+}
+
+.form-group {
+  display: flex;
+  flex-direction: column;
+  gap: 0.35rem;
+}
+
+label {
+  font-size: 0.8rem;
+  font-weight: 500;
+  color: var(--text-muted);
+}
+
+select, input[type="date"] {
+  background: var(--surface2);
+  border: 1px solid var(--border);
+  border-radius: var(--radius);
+  color: var(--text);
+  padding: 0.55rem 0.75rem;
+  font-size: 0.9rem;
+  width: 100%;
+  appearance: none;
+  cursor: pointer;
+  transition: border-color 0.15s;
+}
+
+select:focus, input[type="date"]:focus {
+  outline: none;
+  border-color: var(--accent);
+}
+
+select:disabled, input:disabled {
+  opacity: 0.5;
+  cursor: not-allowed;
+}
+
+/* ── Buttons ── */
+.btn {
+  display: inline-flex;
+  align-items: center;
+  gap: 0.4rem;
+  padding: 0.55rem 1.25rem;
+  border-radius: var(--radius);
+  font-size: 0.9rem;
+  font-weight: 600;
+  cursor: pointer;
+  border: none;
+  transition: background 0.15s, opacity 0.15s;
+  text-decoration: none;
+}
+
+.btn-primary {
+  background: var(--accent);
+  color: #0f172a;
+}
+.btn-primary:hover { background: var(--accent-hover); }
+.btn-primary:disabled { opacity: 0.5; cursor: not-allowed; }
+
+.btn-outline {
+  background: transparent;
+  border: 1px solid var(--border);
+  color: var(--text);
+}
+.btn-outline:hover { border-color: var(--accent); color: var(--accent); }
+
+/* ── Status & alerts ── */
+.alert {
+  padding: 0.85rem 1.1rem;
+  border-radius: var(--radius);
+  font-size: 0.875rem;
+  display: flex;
+  gap: 0.6rem;
+  align-items: flex-start;
+}
+.alert-error { background: rgba(239,68,68,0.15); border: 1px solid rgba(239,68,68,0.4); color: #fca5a5; }
+.alert-info  { background: rgba(56,189,248,0.1);  border: 1px solid rgba(56,189,248,0.3); color: var(--accent); }
+
+.hidden { display: none !important; }
+
+/* ── Loading spinner ── */
+.spinner-wrap {
+  display: flex;
+  flex-direction: column;
+  align-items: center;
+  gap: 1rem;
+  padding: 3rem;
+  color: var(--text-muted);
+  font-size: 0.875rem;
+}
+
+.spinner {
+  width: 36px; height: 36px;
+  border: 3px solid var(--border);
+  border-top-color: var(--accent);
+  border-radius: 50%;
+  animation: spin 0.8s linear infinite;
+}
+@keyframes spin { to { transform: rotate(360deg); } }
+
+/* ── Summary cards ── */
+.stats-grid {
+  display: grid;
+  grid-template-columns: repeat(auto-fit, minmax(140px, 1fr));
+  gap: 1rem;
+  margin-bottom: 1.25rem;
+}
+
+.stat-card {
+  background: var(--surface2);
+  border: 1px solid var(--border);
+  border-radius: var(--radius);
+  padding: 1rem;
+  text-align: center;
+}
+
+.stat-value {
+  font-size: 1.4rem;
+  font-weight: 700;
+  color: var(--accent);
+}
+
+.stat-label {
+  font-size: 0.72rem;
+  color: var(--text-muted);
+  text-transform: uppercase;
+  letter-spacing: 0.04em;
+  margin-top: 0.2rem;
+}
+
+/* ── Quality badge strip ── */
+.quality-strip {
+  display: flex;
+  flex-wrap: wrap;
+  gap: 0.6rem;
+  margin-bottom: 1.25rem;
+}
+
+.q-badge {
+  display: inline-flex;
+  align-items: center;
+  gap: 0.4rem;
+  padding: 0.35rem 0.7rem;
+  border-radius: 999px;
+  font-size: 0.8rem;
+  font-weight: 500;
+}
+.q-badge .dot {
+  width: 8px; height: 8px;
+  border-radius: 50%;
+  flex-shrink: 0;
+}
+
+/* ── Chart ── */
+#chart-container {
+  width: 100%;
+  height: 400px;
+}
+
+/* ── Download bar ── */
+.download-bar {
+  display: flex;
+  gap: 0.75rem;
+  align-items: center;
+  flex-wrap: wrap;
+}
+
+.download-bar span {
+  font-size: 0.8rem;
+  color: var(--text-muted);
+}
+
+/* ── Table ── */
+.table-wrap {
+  overflow-x: auto;
+  max-height: 400px;
+  border: 1px solid var(--border);
+  border-radius: var(--radius);
+}
+
+table {
+  width: 100%;
+  border-collapse: collapse;
+  font-size: 0.82rem;
+}
+
+thead {
+  position: sticky;
+  top: 0;
+  background: var(--surface2);
+  z-index: 1;
+}
+
+th {
+  padding: 0.6rem 0.8rem;
+  text-align: left;
+  font-weight: 600;
+  color: var(--text-muted);
+  border-bottom: 1px solid var(--border);
+  white-space: nowrap;
+}
+
+td {
+  padding: 0.45rem 0.8rem;
+  border-bottom: 1px solid rgba(51,65,85,0.5);
+  white-space: nowrap;
+}
+
+tr:last-child td { border-bottom: none; }
+tr:hover td { background: rgba(255,255,255,0.03); }
+
+.flag-good    { color: var(--good); }
+.flag-suspect { color: var(--suspect); }
+.flag-bad     { color: var(--bad); }
+.flag-na      { color: var(--na); }
+
+.table-note {
+  font-size: 0.75rem;
+  color: var(--text-muted);
+  margin-top: 0.5rem;
+}
+
+/* ── Footer ── */
+footer {
+  text-align: center;
+  padding: 2rem;
+  font-size: 0.78rem;
+  color: var(--text-muted);
+  border-top: 1px solid var(--border);
+}
+
+footer a { color: var(--accent); text-decoration: none; }
+footer a:hover { text-decoration: underline; }
+
+@media (max-width: 600px) {
+  header { flex-direction: column; align-items: flex-start; }
+  .controls { grid-template-columns: 1fr; }
+}

app/static/index.html

@@ -0,0 +1,152 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="UTF-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+  <title>NEM Battery SCADA Data Explorer</title>
+  <link rel="preconnect" href="https://fonts.googleapis.com" />
+  <link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet" />
+  <link rel="stylesheet" href="/static/css/style.css" />
+  <script src="https://cdn.plot.ly/plotly-2.27.0.min.js" defer></script>
+</head>
+<body>
+
+<!-- ── Header ── -->
+<header>
+  <div>
+    <h1>🔋 NEM Battery SCADA Data Explorer</h1>
+    <p>4-second SCADA data for Battery Energy Storage Systems in Australia's National Electricity Market</p>
+  </div>
+</header>
+
+<main>
+
+  <!-- ── Control panel ── -->
+  <section class="panel">
+    <h2>Select Data</h2>
+    <div class="controls">
+
+      <div class="form-group">
+        <label for="sel-state">State / Region</label>
+        <select id="sel-state" disabled>
+          <option value="">— Loading… —</option>
+        </select>
+      </div>
+
+      <div class="form-group">
+        <label for="sel-bess">Battery (BESS)</label>
+        <select id="sel-bess" disabled>
+          <option value="">— Select a state first —</option>
+        </select>
+      </div>
+
+      <div class="form-group">
+        <label for="inp-date">Date <span style="color:var(--text-muted);font-weight:400">(max 1 day)</span></label>
+        <input type="date" id="inp-date" />
+      </div>
+
+      <div class="form-group">
+        <label>&nbsp;</label>
+        <button class="btn btn-primary" id="btn-load" disabled>Load Data</button>
+      </div>
+
+    </div>
+  </section>
+
+  <!-- ── Error ── -->
+  <div id="error-box" class="alert alert-error hidden" role="alert">
+    ⚠️ <span id="error-msg"></span>
+  </div>
+
+  <!-- ── Loading ── -->
+  <div id="loading-box" class="hidden">
+    <div class="spinner-wrap">
+      <div class="spinner"></div>
+      <p>Fetching data from AEMO NEMWEB… this may take 10–20 seconds for large files.</p>
+    </div>
+  </div>
+
+  <!-- ── Results ── -->
+  <div id="results-box" class="hidden">
+
+    <!-- Summary stats -->
+    <section class="panel">
+      <h2>Summary Statistics</h2>
+      <div id="stats-grid" class="stats-grid"></div>
+
+      <h2 style="margin-bottom:0.6rem">Data Quality</h2>
+      <div id="quality-strip" class="quality-strip"></div>
+      <p style="font-size:0.78rem;color:var(--text-muted)">
+        Quality flags are sourced directly from the AEMO FPP dataset.
+        <strong style="color:var(--good)">Good (0)</strong> = reliable measurement;
+        <strong style="color:var(--suspect)">Suspect (1)</strong> = possible communication issue;
+        <strong style="color:var(--bad)">Bad (2)</strong> = sustained failure or substituted value.
+      </p>
+    </section>
+
+    <!-- Chart -->
+    <section class="panel">
+      <h2>Power Output Over Time</h2>
+      <div id="chart-container"></div>
+    </section>
+
+    <!-- Download + table -->
+    <section class="panel">
+      <h2>Download Full Dataset</h2>
+      <div class="download-bar" style="margin-bottom:1.25rem">
+        <span>Download all rows for the selected BESS and date:</span>
+        <a id="btn-csv"     class="btn btn-outline" href="#" download>⬇ CSV</a>
+        <a id="btn-parquet" class="btn btn-outline" href="#" download>⬇ Parquet</a>
+      </div>
+      <p style="font-size:0.78rem;color:var(--text-muted);margin-bottom:1.25rem">
+        <strong>Parquet</strong> is recommended for Python users — it is 5–10× smaller than CSV and
+        loads natively with <code>pandas.read_parquet()</code> or <code>polars.read_parquet()</code>.
+      </p>
+
+      <h2 style="margin-bottom:0.6rem">Data Preview</h2>
+      <div class="table-wrap">
+        <table>
+          <thead>
+            <tr>
+              <th>Measurement Time</th>
+              <th>Market Interval</th>
+              <th>Measured MW</th>
+              <th>Quality Flag</th>
+            </tr>
+          </thead>
+          <tbody id="table-body"></tbody>
+        </table>
+      </div>
+      <p id="table-note" class="table-note hidden"></p>
+    </section>
+
+  </div><!-- /results -->
+
+  <!-- ── Info ── -->
+  <section class="panel">
+    <h2>About This App</h2>
+    <p style="font-size:0.875rem;color:var(--text-muted);line-height:1.7">
+      This tool provides access to 4-second SCADA data published by the
+      <a href="https://www.aemo.com.au" target="_blank" rel="noopener">Australian Energy Market Operator (AEMO)</a>
+      via the <a href="https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/" target="_blank" rel="noopener">NEMWEB FPPDAILY</a>
+      report. Data is available from <strong>9 December 2024</strong> when the
+      <a href="https://aemo.com.au/initiatives/major-programs/frequency-performance-payments-project" target="_blank" rel="noopener">Frequency Performance Payments (FPP)</a>
+      scheme commenced non-financial operation.<br/><br/>
+      Data is fetched live from AEMO on each request — no data is stored on this server.
+      Maximum request window: <strong>1 day</strong> per query.
+    </p>
+  </section>
+
+</main>
+
+<footer>
+  Data source: <a href="https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/" target="_blank">AEMO NEMWEB FPPDAILY</a>
+  &nbsp;|&nbsp;
+  BESS list: <a href="https://www.aemo.com.au/energy-systems/electricity/national-electricity-market-nem/planning_and_forecasting/generation-information" target="_blank">AEMO NEM Generation Information</a>
+  &nbsp;|&nbsp;
+  Not affiliated with AEMO.
+</footer>
+
+<script src="/static/js/app.js"></script>
+</body>
+</html>

app/static/js/app.js

@@ -0,0 +1,268 @@
+/* ─────────────────────────────────────────
+   NEM BESS SCADA Explorer — frontend logic
+   ───────────────────────────────────────── */
+
+const API = '';   // same origin
+
+/* ── State ── */
+let bessList = {};
+let qualityFlags = {};
+let currentData = null;
+let currentDuid = null;
+let currentDate = null;
+
+/* ── DOM refs ── */
+const selState   = document.getElementById('sel-state');
+const selBess    = document.getElementById('sel-bess');
+const inpDate    = document.getElementById('inp-date');
+const btnLoad    = document.getElementById('btn-load');
+const errorBox   = document.getElementById('error-box');
+const errorMsg   = document.getElementById('error-msg');
+const loadingBox = document.getElementById('loading-box');
+const resultsBox = document.getElementById('results-box');
+const statsGrid  = document.getElementById('stats-grid');
+const qualStrip  = document.getElementById('quality-strip');
+const tableBody  = document.getElementById('table-body');
+const tableNote  = document.getElementById('table-note');
+const btnCsv     = document.getElementById('btn-csv');
+const btnParquet = document.getElementById('btn-parquet');
+
+/* ── Boot ── */
+async function init() {
+  // Set default date to yesterday
+  const yesterday = new Date();
+  yesterday.setDate(yesterday.getDate() - 1);
+  inpDate.value = yesterday.toISOString().slice(0, 10);
+  inpDate.max   = yesterday.toISOString().slice(0, 10);
+
+  // Fetch app info for min date
+  try {
+    const info = await fetch(`${API}/api/info`).then(r => r.json());
+    inpDate.min = info.data_start_date;
+  } catch (_) {}
+
+  // Load BESS list and quality flags in parallel
+  try {
+    [bessList, qualityFlags] = await Promise.all([
+      fetch(`${API}/api/bess`).then(r => r.json()),
+      fetch(`${API}/api/quality-flags`).then(r => r.json()),
+    ]);
+  } catch (e) {
+    showError('Failed to load BESS list. Please refresh the page.');
+    return;
+  }
+
+  // Populate state selector
+  Object.keys(bessList).sort().forEach(state => {
+    const opt = document.createElement('option');
+    opt.value = state;
+    opt.textContent = state;
+    selState.appendChild(opt);
+  });
+
+  selState.disabled = false;
+  selState.addEventListener('change', onStateChange);
+  btnLoad.addEventListener('click', onLoad);
+  selBess.addEventListener('change', () => { btnLoad.disabled = !selBess.value; });
+}
+
+function onStateChange() {
+  const state = selState.value;
+  selBess.innerHTML = '<option value="">— Select BESS —</option>';
+  selBess.disabled  = !state;
+  btnLoad.disabled  = true;
+
+  if (!state || !bessList[state]) return;
+
+  bessList[state].forEach(b => {
+    const opt = document.createElement('option');
+    opt.value = b.duid;
+    opt.textContent = `${b.name} (${b.capacity_mw} MW)`;
+    selBess.appendChild(opt);
+  });
+}
+
+/* ── Load data ── */
+async function onLoad() {
+  const duid = selBess.value;
+  const date = inpDate.value;
+
+  if (!duid || !date) return;
+
+  hideError();
+  hideResults();
+  showLoading(true);
+  btnLoad.disabled = true;
+
+  try {
+    const resp = await fetch(`${API}/api/data?duid=${encodeURIComponent(duid)}&date=${date}`);
+    if (!resp.ok) {
+      const err = await resp.json().catch(() => ({ detail: 'Unknown error' }));
+      throw new Error(err.detail || `Server error ${resp.status}`);
+    }
+    const payload = await resp.json();
+    currentData  = payload;
+    currentDuid  = duid;
+    currentDate  = date;
+    renderResults(payload, duid, date);
+  } catch (e) {
+    showError(e.message);
+  } finally {
+    showLoading(false);
+    btnLoad.disabled = false;
+  }
+}
+
+/* ── Render ── */
+function renderResults(payload, duid, date) {
+  const { summary, data } = payload;
+
+  renderStats(summary, payload.total_rows);
+  renderQualityStrip(summary.flag_breakdown);
+  renderChart(data);
+  renderTable(data, payload.total_rows);
+
+  const base = `${API}/api/download`;
+  const params = `duid=${encodeURIComponent(duid)}&date=${date}`;
+  btnCsv.href     = `${base}/csv?${params}`;
+  btnParquet.href = `${base}/parquet?${params}`;
+
+  resultsBox.classList.remove('hidden');
+}
+
+function renderStats(summary, totalRows) {
+  statsGrid.innerHTML = '';
+  const stats = [
+    { label: 'Total Readings', value: totalRows.toLocaleString() },
+    { label: 'Min MW',  value: summary.min_mw  != null ? summary.min_mw.toFixed(2)  : '—' },
+    { label: 'Max MW',  value: summary.max_mw  != null ? summary.max_mw.toFixed(2)  : '—' },
+    { label: 'Mean MW', value: summary.mean_mw != null ? summary.mean_mw.toFixed(2) : '—' },
+    { label: 'Std Dev', value: summary.std_mw  != null ? summary.std_mw.toFixed(2)  : '—' },
+  ];
+  stats.forEach(s => {
+    statsGrid.insertAdjacentHTML('beforeend', `
+      <div class="stat-card">
+        <div class="stat-value">${s.value}</div>
+        <div class="stat-label">${s.label}</div>
+      </div>`);
+  });
+}
+
+function renderQualityStrip(breakdown) {
+  qualStrip.innerHTML = '';
+  const order = ['0', '1', '2', '-1'];
+  order.forEach(flag => {
+    const info  = qualityFlags[flag] || { label: `Flag ${flag}`, description: '', color: '#94a3b8' };
+    const entry = breakdown[flag];
+    if (!entry) return;
+    qualStrip.insertAdjacentHTML('beforeend', `
+      <div class="q-badge" title="${info.description}">
+        <span class="dot" style="background:${info.color}"></span>
+        <strong>${info.label}</strong>: ${entry.count.toLocaleString()} readings (${entry.pct}%)
+      </div>`);
+  });
+}
+
+function renderChart(data) {
+  if (!data || data.length === 0) return;
+
+  // Group by quality flag for separate traces (so we can colour-code points)
+  const flagGroups = {};
+  data.forEach(row => {
+    const flag = String(row.MW_QUALITY_FLAG ?? '-1');
+    if (!flagGroups[flag]) flagGroups[flag] = { x: [], y: [] };
+    flagGroups[flag].x.push(row.MEASUREMENT_DATETIME);
+    flagGroups[flag].y.push(row.MEASURED_MW);
+  });
+
+  const traces = Object.entries(flagGroups).map(([flag, pts]) => {
+    const info = qualityFlags[flag] || { label: `Flag ${flag}`, color: '#94a3b8' };
+    return {
+      x: pts.x,
+      y: pts.y,
+      mode: 'lines+markers',
+      marker: { size: 2, color: info.color },
+      line: { width: 1, color: info.color },
+      name: info.label,
+      type: 'scattergl',
+    };
+  });
+
+  const layout = {
+    paper_bgcolor: 'transparent',
+    plot_bgcolor:  'transparent',
+    font: { color: '#e2e8f0', size: 11 },
+    xaxis: {
+      title: 'Time',
+      gridcolor: '#334155',
+      linecolor: '#334155',
+      tickfont: { color: '#94a3b8' },
+    },
+    yaxis: {
+      title: 'Power (MW)',
+      gridcolor: '#334155',
+      linecolor: '#334155',
+      tickfont: { color: '#94a3b8' },
+      zeroline: true,
+      zerolinecolor: '#475569',
+    },
+    legend: { orientation: 'h', y: -0.2 },
+    margin: { t: 20, r: 20, b: 60, l: 60 },
+    hovermode: 'x unified',
+  };
+
+  Plotly.react('chart-container', traces, layout, {
+    responsive: true,
+    displayModeBar: true,
+    modeBarButtonsToRemove: ['select2d', 'lasso2d'],
+  });
+}
+
+function flagClass(flag) {
+  switch (String(flag)) {
+    case '0':  return 'flag-good';
+    case '1':  return 'flag-suspect';
+    case '2':  return 'flag-bad';
+    default:   return 'flag-na';
+  }
+}
+
+function flagLabel(flag) {
+  return (qualityFlags[String(flag)] || { label: flag }).label;
+}
+
+function renderTable(data, totalRows) {
+  tableBody.innerHTML = '';
+  data.forEach(row => {
+    const cls = flagClass(row.MW_QUALITY_FLAG);
+    tableBody.insertAdjacentHTML('beforeend', `
+      <tr>
+        <td>${row.MEASUREMENT_DATETIME ?? ''}</td>
+        <td>${row.INTERVAL_DATETIME ?? ''}</td>
+        <td>${row.MEASURED_MW != null ? row.MEASURED_MW.toFixed(4) : '—'}</td>
+        <td class="${cls}">${flagLabel(row.MW_QUALITY_FLAG)}</td>
+      </tr>`);
+  });
+
+  if (totalRows > data.length) {
+    tableNote.textContent =
+      `Showing ${data.length.toLocaleString()} of ${totalRows.toLocaleString()} rows. Download the full dataset using the buttons above.`;
+    tableNote.classList.remove('hidden');
+  } else {
+    tableNote.classList.add('hidden');
+  }
+}
+
+/* ── UI helpers ── */
+function showError(msg) {
+  errorMsg.textContent = msg;
+  errorBox.classList.remove('hidden');
+}
+function hideError() { errorBox.classList.add('hidden'); }
+function showLoading(show) {
+  loadingBox.classList.toggle('hidden', !show);
+}
+function hideResults() { resultsBox.classList.add('hidden'); }
+
+/* ── Start ── */
+init();

requirements.txt

@@ -0,0 +1,7 @@
+fastapi==0.115.5
+uvicorn[standard]==0.32.1
+polars==1.15.0
+httpx==0.27.2
+pyarrow==18.1.0
+python-multipart==0.0.12
+aiofiles==24.1.0