PLAN.md

BESS SCADA Data Web App — Implementation Plan

Architecture Overview

Hosting: HuggingFace Spaces (Docker SDK) Backend: FastAPI (Python) Frontend: HTML/CSS/JS served by FastAPI (lightweight, no React build step needed) Data Processing: Polars (memory-efficient, faster than pandas for large CSVs) Charts: Plotly.js (interactive, zoom/pan, rendered client-side) Analytics: Lightweight SQLite logging (ephemeral but rebuilt; optionally push to free Supabase) BESS List: Static JSON file in repo, manually updated monthly

Data Flow

Student selects: State → BESS → Date
         ↓
Frontend sends request to FastAPI: GET /api/data?duid=XXXX&date=2026-03-01
         ↓
Backend constructs AEMO URL:
  - Current: https://www.nemweb.com.au/REPORTS/Current/FPPDAILY/
  - Archive: https://nemweb.com.au/Reports/Archive/FPPDAILY/
         ↓
Backend downloads ZIP from AEMO (~20-50 MB compressed)
         ↓
Backend extracts CSV in-memory (streaming, never full file to disk)
         ↓
Backend filters rows: FPP_UNITID == selected DUID
  Result: ~21,600 rows (~1-2 MB) for one BESS for one day
         ↓
Backend returns JSON (for chart/table display) or CSV/Parquet (for download)
         ↓
Frontend renders interactive chart + paginated table

Project Structure

BESS-SCADA-Data/
├── Dockerfile                  # HuggingFace Spaces Docker config
├── requirements.txt            # Python dependencies
├── README.md                   # HuggingFace Spaces metadata
├── app/
│   ├── main.py                 # FastAPI application entry point
│   ├── config.py               # Constants, URLs, settings
│   ├── data/
│   │   ├── bess_list.json      # Static BESS list by state (updated monthly)
│   │   └── quality_flags.json  # MW_QUALITY_FLAG descriptions
│   ├── services/
│   │   ├── aemo_fetcher.py     # Download & extract ZIP from AEMO
│   │   ├── data_processor.py   # Filter, transform CSV data using Polars
│   │   └── analytics.py        # Log requests (IP, timestamp, BESS, date)
│   ├── routers/
│   │   ├── api.py              # REST API endpoints
│   │   └── pages.py            # Serve HTML pages
│   └── static/
│       ├── index.html          # Main single-page app
│       ├── css/
│       │   └── style.css       # Styling
│       └── js/
│           └── app.js          # Frontend logic (fetch, render chart/table)

Implementation Steps

Step 1: Project Setup & Docker Configuration

• Initialize project structure
• Create Dockerfile for HuggingFace Spaces (Python 3.11 slim, expose port 7860)
• Create requirements.txt: fastapi, uvicorn, polars, httpx, pyarrow
• Create HuggingFace Spaces README.md with metadata (sdk: docker)

Step 2: Static BESS List

• Parse the AEMO NEM Generation Information Excel manually
• Create bess_list.json with structure:

  {
    "NSW": [
      {"duid": "BESS1", "name": "Battery Name", "capacity_mw": 100, "region": "NSW1"}
    ],
    "VIC": [...],
    "QLD": [...],
    "SA": [...],
    "TAS": [...]
  }
  

• Create quality_flags.json:

  {
    "0": {"label": "Good", "description": "Good quality data", "color": "green"},
    "1": {"label": "Bad", "description": "Sustained communication failure or manual override", "color": "red"},
    "-1": {"label": "N/A", "description": "Not applicable", "color": "gray"}
  }
  

Step 3: AEMO Data Fetcher Service

• aemo_fetcher.py:
  • Given a date, construct the correct AEMO URL (Current vs Archive)
  • Download ZIP using httpx with 30-second timeout
  • Extract CSV from ZIP in-memory using zipfile module
  • Return raw CSV bytes/stream
  • Error handling: timeout → retry once → user-friendly error
  • Need to investigate the exact filename pattern in the ZIP to construct URLs correctly

Step 4: Data Processor Service

• data_processor.py:
  • Read CSV bytes with Polars (lazy mode for memory efficiency)
  • Filter by FPP_UNITID == selected_duid
  • Select only relevant columns: INTERVAL_DATETIME, MEASUREMENT_DATETIME, FPP_UNITID, MEASURED_MW, MW_QUALITY_FLAG
  • Return as:
    • Polars DataFrame (for JSON serialization to frontend)
    • CSV bytes (for CSV download)
    • Parquet bytes (for Parquet download)
  • Compute summary statistics: min, max, mean MW, % good quality, % bad quality

Step 5: Analytics Service

• analytics.py:
  • On each data request, log to SQLite (/tmp/analytics.db):
    • Timestamp
    • IP address (from request headers, respecting X-Forwarded-For)
    • Selected BESS (DUID)
    • Selected date
    • Download format (view/csv/parquet)
  • Provide endpoint to view analytics (protected by simple query param token)
  • Note: SQLite on ephemeral storage will be lost on restart — acceptable for basic tracking. Could optionally push to free Supabase (500 MB free) for persistence.

Step 6: FastAPI Backend Endpoints

• GET / — Serve main HTML page
• GET /api/bess — Return BESS list grouped by state
• GET /api/data?duid={duid}&date={YYYY-MM-DD} — Return filtered SCADA data as JSON
  • Response includes: data rows, summary stats, quality flag breakdown
• GET /api/download/csv?duid={duid}&date={YYYY-MM-DD} — Download as CSV
• GET /api/download/parquet?duid={duid}&date={YYYY-MM-DD} — Download as Parquet
• GET /api/analytics?token={secret} — View usage analytics (admin only)

Step 7: Frontend (Single Page App)

• State selector: Dropdown for Australian state (NSW, VIC, QLD, SA, TAS)
• BESS selector: Populated dynamically based on selected state
• Date picker: Single date (max 1 day), with validation (not future dates, not before data availability)
• Load button: Fetches data from backend
• Loading state: Spinner with message "Fetching data from AEMO... this may take 10-15 seconds"
• Chart: Plotly.js time-series chart of MEASURED_MW vs MEASUREMENT_DATETIME
  • Color-coded by MW_QUALITY_FLAG (green=good, red=bad, gray=N/A)
  • Interactive zoom, pan, hover tooltips
• Data quality banner:
  • "X% of measurements are good quality, Y% flagged as bad quality (communication failure or manual override)"
• Data table: Paginated table showing all rows (virtual scrolling for performance)
• Download buttons: CSV and Parquet download buttons
• Summary stats panel: Min, Max, Mean, Std Dev of MEASURED_MW

Step 8: Error Handling & UX

• AEMO server unavailable → "AEMO data source is temporarily unavailable. Please try again in a few minutes."
• No data for selected BESS/date → "No SCADA data found for [BESS name] on [date]. This unit may not have been operational on this date."
• Large response handling → Stream response, show progress
• Rate limiting → Simple in-memory rate limit (10 requests/minute per IP) to prevent abuse

Step 9: Deployment to HuggingFace Spaces

• Create HuggingFace Space with Docker SDK
• Push code to HuggingFace repo
• Configure Space settings
• Test with sample BESS queries

Step 10: Documentation & BESS List Maintenance

• Document how to update bess_list.json when AEMO publishes new generation info
• Provide a helper script to parse the AEMO Excel and generate the JSON

Key Technical Decisions

Decision	Choice	Rationale
Polars over Pandas	Polars	2-10x faster, much lower memory usage, lazy evaluation
FastAPI over Flask	FastAPI	Async support (critical for downloading from AEMO), auto-docs, modern
Plotly.js over server-side charts	Plotly.js	Client-side rendering reduces server load, interactive
Single HTML page over React	Vanilla JS	No build step, simpler deployment, fewer dependencies
Parquet for download	PyArrow	Parquet is 5-10x smaller than CSV, native to Python data science workflows
httpx over requests	httpx	Async support, connection pooling, timeout handling

Risks & Mitigations

Risk	Mitigation
AEMO changes URL structure	Configurable URLs in config.py, easy to update
HF Space sleeps after inactivity	Acceptable for academic use; wakes in ~30 seconds
16 GB RAM exceeded	On-demand fetch + Polars lazy mode keeps memory under 500 MB per request
AEMO rate-limits our requests	Cache recently fetched data in /tmp (ephemeral but helps during active sessions)
BESS list becomes stale	Monthly manual update process documented; helper script provided